Web Design and Development
Drupal 8.1.3 and 7.44, maintenance releases which contain fixes for security vulnerabilities, are now available for download.Download Drupal 8.1.3 Download Drupal 7.44
Upgrading your existing Drupal 8 and 7 sites is strongly recommended. There are no new features or non-security-related bug fixes in these releases. For more information about the Drupal 8.1.x release series, consult the Drupal 8 overview. More information on the Drupal 7.x release series can be found in the Drupal 7.0 release announcement.Security vulnerabilities
Drupal 8.1.3 and 7.44 were released in response to the discovery of security vulnerabilities. Details can be found in the official security advisory:
To fix the security vulnerabilities, please upgrade to either Drupal 8.1.3 or Drupal 7.44.Change log
We have a security announcement mailing list and a history of all security advisories, as well as an RSS feed with the most recent security advisories. We strongly advise Drupal administrators to sign up for the list.
Drupal 8 and 7 include the built-in Update Manager module, which informs you about important updates to your modules and themes.Bug reports
The goal of automated testing is confidence: confidence in application stability, and confidence that new features work as intended. Continuous integration as a philosophy is about speeding the rate of change while keeping stability. As the number of contributing programmers increase, the need to have automated testing as a means to prove stability increases.
This post is focused on how the automated testing infrastructure on Drupal.org works, not actually writing tests. Much more detail about how to write tests during Drupal development can be found in community documentation:
DrupalCI essentially runs two categories of tests:
Functional tests (also called blackbox testing) are the most common type of test run on DrupalCI hardware. These tests run assertions that test functionality by installing Drupal with a fresh database and then exercising that installation by inserting data and confirming the assertions complete. Front-end tests and behavior driven tests (BDD) tend to be functional. Upgrade tests are a type of functional tests that run a full installation of Drupal, then run upgrade commands.
Unit tests run assertions that test a unit of code and do not require a database installation. This means they execute very quickly. Because of its architecture, Drupal 8 has much more unit test coverage than Drupal 7.
These test categories can be broken down further into more specific test types.What testing means at the scale of Drupal
Drupal 8, with its 3,000+ core contributors and 7,288 contrib developers (so far), needs testing as a means to comfortably move forward code that everyone can trust to be stable.
Between January and May 2016, 90,364 test runs were triggered in DrupalCI. That is about 18,000 test runs requested per month. Maintainers set whether they want tests to run on demand, with every patch submitted, or nightly. They also determine what environments those tests will run on; there are 6 combinations of PHP and database engines available for maintainers to choose from.
The majority of these test runs are Drupal 8 tests at this point. (19,599 core tests and 47,713 contrib project tests were run during those 5 months.) Each test costs about 12 cents to run on Amazon Web Services. At the time of writing this post, we averaged around $2,000 per month in testing costs for our community. (Thank you supporters!)An overly simple history of automated testing for Drupal
Automated testing first became a thing for Drupal contributed projects Drupal 4.5 with the introduction of the SimpleTest module. It was not until Drupal 6 that we started manually building out testbots and running these tests on Drupal.org hardware.
In Drupal 7, SimpleTest was brought into Drupal Core. (More information about what that took can be reviewed in the SimpleTest Roadmap for Drupal 7.)
In Drupal 8, PHPUnit testing was added to Drupal Core. PHPUnit tests are much faster than a full functional test in SimpleTest—though runtest.sh still triggers a combination of these test types in Drupal 8.
The actual implementation of automated testing was much more complicated that this history suggests. The original testbot infrastructure that ran for 7 years on Drupal.org hardware was manually managed by some fiercely dedicated volunteers. The manual nature of that maintenance led to the architecture of DrupalCI, which was meant to make it easier to test locally at first and later focused on autoscaling on powerful hardware that could plow through tests more quickly.DrupalCI's basic structure
In The Drupal.org Complexity, we could see the intricate ways that Drupal's code base interacts with other parts of the system.
We could further break out how systems like DrupalCI are interrelated.
DrupalCI is a combination of data stored on Drupal.org, cron jobs, drush commands, and most importantly a couple of Jenkins installations to manage all the automation.
Jenkins is the open source automation server project that makes most of the system possible. We use it for automating our build process and deploying code to our dev, staging and production environments. It automates just about anything and is used by companies small and large to run continuous integration or continuous deployment for their applications. It's considered a "best practice" solution alongside options like Travis, CircleCI, and Bamboo. They all have slightly different features, but automation is at the core of most of these DevOps tools.
To provide continuously integrated tests, you need to trigger those tests at a moment when the tests will have the greatest value.
The three triggers for running a test job are when a patch is added to an issue comment, when code is committed to a repository or daily on a cron. Maintainers can specify which triggers are associated with which branches of their projects and which environments should run those tests.
For core these settings look something like this:
This detail allows for specific tests to run at specific times per the Drupal.org Testing Policy for DrupalCI.
To make this automation happen, we have an installation of Jenkins (Infrastructure Jenkins below) that is polled by Drupal.org once per minute with testing jobs to be queued.
These jobs live in a database record alongside Drupal.org.
Infrastructure Jenkins speaks to the CI Dispatcher (also Jenkins) where the testing queue regularly passes those jobs to available testbots. CI Dispatcher using an Amazon Web Services EC2 plugin to spin up new testbots when no existing testbot is available. Each testbot is a able to spin up Docker containers with the specific test images defined by the maintainer. Theses containers pull from DockerHub repositories with official combinations of PHP and database engines that Drupal supports.
After a testbot is running, the CI Dispatcher is in constant communication with the bots. You can even click through to the console on CI Dispatcher and watch the tests run. (It's not very exciting—perhaps we should add sound effects to the failures—but it is very handy.)
Once per minute, Drupal.org is polling the CI Dispatcher for test status. It responds with pending, running, failed or passed. Failed and passed tests are then pulled back into Drupal.org for display using the Jenkins JSON API.
Tests can also be run on demand at the patch, commit or branch level using the handy "add test" and "retest" links.Why did we build this ourselves? Why not use [insert testing platform here]
Lot's of people have asked "why don't we use TravisCI, CircleCI or some other hosted testing solution." The short answer is that most publicly available testing systems require Github authentication and integration.
Additionally, our testing infrastructure is powerful because of its integration with our issue queues. Read the aforementioned The Drupal.org Complexity for more information.
Another reason to run our own testing is scale. To get through all of the core tests for Drupal 8 in an acceptable amount of time (about 44 mins on average), we run very large testbots. These bots have 2 processors with 8 hardware cores. With hyperthreading, that means we have 32 hardware execution threads—about 88 EC2 compute units. They are not exactly super computers, but they are very performant.
We average nearly 18,000 test runs per month. During our peak usage we spin up as many as 25 testbot machines—though usually we cap at 15 bots to keep costs under control. This helps us plow through our testing needs during sprints at DrupalCons and large camps.
We have explored using an enterprise licensed version of either Github or CircleCI with our own hardware to tackle testing. That same consideration has been given to SauceLabs for front-end testing. Right now, there is not a cost savings to tackle this migration, but that does not rule it out in the future. Testing services continues to evolve.Accelerating Drupal 8
In my first months as CTO, I was told repeatedly that the most important thing for us to work on was testing for Drupal 8. In those early days as I built out the team, we were mostly focused on catching up from the Drupal 7 upgrade and tackling technical debt issues that cropped up. In DrupalCon Austin, I had members of my team learn how to maintain the testbot infrastructure so that we could take over the process of spinning up bots and dealing with spikes in demand.
By early 2015, we had optimized the old testbots as much as they were going to be optimized. We moved them to AWS so we could spin up faster machines and more bots, but there were features that were waiting on the new DrupalCI infrastructure that were blocking key development on Drupal 8.
In March of 2015, we invited all the community developers that had helped with DrupalCI to the Drupal Association offices in Portland and sprinted with them to figure out the remaining implementation needs. The next couple of months involved tweaking DrupalCI's architecture and cutting out any nice to have features to get something launched as soon as possible.
It is no coincidence that from the time of DrupalCI's launch until the release of Drupal 8, progress was rapidly accelerated.
I am immensely proud of the work of all the community members and staff that worked directly with core maintainers to unblock Drupal 8 development and make it faster. This work was critical.
Thank you to jthorson, ricardoamaro, nick_schuch, dasrecht, isntall, drumm, mikey_p, mixologic, hestenet, chx, mile23, alexpott, dawehner, Shyamala, and webchick. You all made DrupalCI. (And huge apologies to all those I'm undoubtedly leaving out.) Also thank you to anyone who chimed in on IRC or in the issue queues to help us track down bugs and improve the service.What's next for testing Drupal
Most of the future state of testing is outlined in the Drupal.org Testing Policy for DrupalCI.
Key issues that we still need to solve are related to concurrent testing improvements and new test types to support. While we have PhantomJS integrated with the test runner, there are optimizations that need to happen.
Testing is not an endpoint. Like much of our work, it is an ongoing effort to continuously improve Drupal by providing a tool that improves how we test, what we test, and when we test.
Matthew Lechleider (Slurpee) has been active in the Drupal community for over a decade, and his hard work has directly led to an incredible amount of community growth. The founder of a Chicago Drupal User Group and our community’s chief advocate for the Google Summer of Code and Google Code-In programs, Matthew has been a key part of growing the Drupal project and our global community. Here's his Drupal story.
“In 2005, I was a full-time university student working at an internet service provider so I could put myself through school,” Matthew said. “I was working as a network/systems person, and since I was at an ISP we had a lot of people calling us and asking the same questions over and over. At the time, I knew bit about web development and programming, and I thought, ‘I bet I could make a website that would answer these people’s questions.’ And that’s how I found Drupal. I proposed it to my boss, and the next thing I knew I was working on a full-time project getting paid to work with Drupal 4. I built the website and it was really popular— and we noticed that the phone calls went down. We were tracking our support calls at the 24-hour call center, and when people called for help, we would refer them to the website as a resource. So it really was a big help."
After that, the next steps were logical for Matthew. He put together a Drupal meet-up at his Chicago-based company. The group grew quickly each month, and in no time at all, people were asking about training and “Introduction to Drupal” classes. "I started teaching those classes,” Matthew said, "and then next thing you know, people were asking for private trainings and businesses were asking me to come to their offices and train new Drupal developers. When the people I was training came back with advanced questions, I realized how much money they were making, so in 2008 I went from being a network engineer to focusing on Drupal full-time. Since then, I’ve started a Drupal business and worked on some very big projects."
"I never thought I would be a web developer, but I fell into Drupal, saw how great and easy it was, and decided it was a good thing to be a part of,” Matthew added.
Over his time in Drupal, Matthew has converted a lot of Chicagoan web developers into Drupal users. “It's pretty cool to be part of something bigger than yourself,” Matthew said. “It's like a big tidal wave — I feel like I’ve been riding this Drupal wave for a long time. I didn’t think I’d still be work with Drupal this many years later."Why Slurpee?
Many people in the community know Matthew only by his user username, Slurpee. But how did he come by that handle?
"I was probably eight or nine years old, learning about computers, and I had some nicknames I was playing around with. But it’s like that movie ‘Hackers’: you have to have your handle, you have to have your identity. It was the middle of a hot July in the summer, and as I was figuring out what I should call myself, I realized I had bout 20 empty slurpee cups surrounding my computer. I really do like slurpees. So that’s where that came from."Drupal 8
As a long-time Drupal user and evangelist, Matthew is incredibly excited for Drupal 8.
" I have a traditional programming background in computer science, and Drupal wasn’t always the most professional CMS,” Matthew said. “Now, I’m very excited about Drupal 8— it’s like Drupal grew up and went to college and got a graduate degree. Drupal 8 is the way Drupal should have been a long time ago. I’ve built some of the biggest Drupal projects in the world, and when you’re talking to those kinds of massive clients, it’s hard to sell systems like Drupal four, five, and six. They’re out paying the big money for the huge enterprise solutions, and Drupal 8 is big. It’s ready to go, and I really think it's on par with everything else these days."
When asked about his favorite Drupal 8 feature, Matthew said, “As I work with big sites, it’s been a big struggle to deal with continuous integration, which is resolved in Drupal 8. As a person with a sysadmin background, I think integration is probably the thing that will save the most headaches. It’s going to be a very cool tool to work with."Google Summer of Code and Google Code-In
Matthew is also heavily involved in several of Google’s student coding programs, and runs point on keeping Drupal in the Google Summer of Code (GSoC) and Google Code-In (GCI) programs.
“I’m a bit younger than other people in the community, and I started with Drupal when I was 19 years old or so. I was going to college when I got into Drupal, and I remember talking to people about it at school... and nobody had any clue about it, not even my teachers,” Matthew said. “Fast forward several years: I was working in the community, specifically on the VoIP Drupal module suite. At the time, we had a student come to us about GSoC —this was in 2012— and he said, ‘hey, I want to work on this project for you guys as part of the GSoC. Can you be a mentor?’ I said sure, and that’s all I was that first year. The next year, nobody wanted to organize GSoC for Drupal, so I stepped up and said that I liked the program and would get the ball rolling. I spent a lot of time revamping the things that Drupal does with Google, got us accepted back into the program, and we’ve been participating every year since, both in the Google Summer of Code, which is for university students, and the Google Code-In, which is for students ages 13 through 17.
“Getting back to what I said earlier about being a student, when nobody knew what Drupal was, it was a bit harder to get involved. Knowing that there’s a program like this in schools around the world, pushing these projects to students, I think it’s critical for us to participate. If we want this project to continue to be successful, we have to focus on younger people. They’re the ones who are adapting and changing the tech as we know it, and if they don’t know about Drupal, they won’t use it. But if we embrace these kids and show them how awesome Drupal can be — we have impressive students doing impressive stuff— it’s great for everyone. That’s why I think it’s super important that we spend so much time on the GSoC and GCI programs."
For those who are interested in getting involved, there are both GSoC and GCI Drupal Groups, a documentation guide for GCI, and a guide for the GSoC students who are just getting started. There is also a #drupal-google IRC channel on free node.
“It’s fairly easy to get involved,” Matthew said. “If you’re interested in helping, join the groups — we send out lots of updates — or you can can contact me directly. You can find us on IRC, or if you know of a student who wants to get involved, all they need to do is read the documentation. It covers everything. We spent a lot of time on that documentation,” Matthew said.Being part of something bigger
When asked what drives him to participate and organize the community, Matthew’s answer was simple.
"Honestly, it’s all about being part of something bigger than myself,” he said. "I’ve been participating in computer hacker nerd groups since I was a kid, and back then (in the 90s) it was fairly difficult to find out about these kinds of groups. I attended something called 2600 — does anyone else remember that? — I thought it was so cool to be a part of something like that. I’ve been in IRC every day for over 20 years participating in communities, and it’s so exciting to be part of a huge thing bigger than myself. Now I’m getting recognized for it in Drupal, which is cool,” he added.
"Drupal is by far the largest community I’ve participated in from that point of view, and it’s been an exciting ride. Honestly, I thought it would have been over by now — I’ve been part of the community for over 10 years and that’s a long time — but I’m excited to see where it goes. For me, Drupal is more of a lifestyle than a career. Technically I’m a web developer, but I spend so much of my time volunteering and going to events. I can go to a DrupalCon in another country and see my friends from all over the world. Seeing how mature our community has become over the past decade continues to excite me, and and I plan to be part of this for as long as possible."
“The money’s not bad either,” he added. “I live a comfortable life working from home. I like to travel a lot, go to music festivals, or go on Phish tour for weeks at a time, but I can still work. I have a mobile hotspot, and just bring my laptop with me. I can’t tell you how many times I’ve been set up at a camping spot in the middle of nowhere, working on my laptop, or how many times I’ve been in a random country looking for internet so I can get some work done.”
“Working remotely, or working from home, gives me a lot of freedom. It lets me go to the skate park when all the kids are at school— I skateboard, and bring my board with me when I travel — and there’s time for me to go running every day. It’s actually one of my favorite activities, my running break — halfway through the day, I get up and I go running. It helps me keep my sanity."
"I also like to snowboard,” Matthew continued. “I’ve been skateboarding since I was 5 — since I could walk and talk I’ve been skateboarding — and I started snowboarding a couple years after that. Going to DrupalCon Denver was exciting, and there was a DrupalCon snowboard trip. Lots of people went to Breckenridge together, and that’s the other cool thing about the community: these people are fun, they like to go out and do things together."Helping camps and communities around the world
After a decade in the Drupal community, Matthew has a lot of great memories of traveling, teaching, and sharing.
“I was invited to organize the first DrupalCamp in Sri Lanka and hold training classes there,” Matthew said. “It’s an island just off the coast of India, and when I went in 2012, they had just finished having a civil war. I was one of the only foreigners there, and there was military presence walking around. I wasn’t allowed to leave my hotel unless the Drupal people came and picked me up, and they were absolutely wonderful. They were so nice and appreciative, and so many people were incredibly smart and really talented. After I did a training or presentation, they all had a million questions and we ended up talking for hours. And that was when I had an ‘aha!’ moment and really felt like I was part of something bigger. It drove home that it’s my responsibility to spread the word of Drupal as much as possible."
“That was actually one of the trainings that I did when I was traveling a lot” Matthew added. “I literally traveled for years living out of hostels, and couch surfing. I was being contracted at the time to go teach Drupal in Europe, Asia, and Australia."
And for Matthew, one of his proudest accomplishments is Drupal’s participation in the GSoC and GCI programs.
"I think it’s cool that I maintain Drupal’s relationship with Google,” he said. "It’s Google. They’re the biggest tech company ever, and I have this relationship with multiple people at their offices. They’re all super nice and it’s great that they’re pushing this open source stuff. They want our community's feedback, and I think it’s really cool that I can represent Drupal with Google. They’re trying to give us money and get us to do more with the students. It’s something I think is really great."The importance of networking
When asked if he had any advice for new Drupal users, Matthew had several thoughts to share.
“Find your local communities and participate,” he advised. “Go to as many events as you can. If you have to drive a couple of hours to go to a Drupal camp, do it. If there isn’t a local community in your area, start one. If you go to meetup.com and post a similar topic, even if just one other person shows up, that’s cool and the community will grow. Drupal is a software, but it helps to get some face time in with other people. Network. Go out. Have some coffee (or beer). Hang out. Bring your laptop and start sharing ideas— you’ll be surprised at how welcoming the Drupal community is. I can’t tell you how many times I’ve sat at the bar until the bar kicked us out just helping people fix websites and giving free training.
“If you’re struggling, it’s important to network with other groups — non-Drupal groups — like a web-dev group or an open source group or a Linux group or something. That’s what helped me when I did the first Drupal camp for Chicago 2008. We were thinking 100 people might show up, but we did a lot of networking. I contacted every tech user group in Milwaukee, Detroit, southern Illinois, and Minnesota. I found every group that was related to web or open source or Drupal, and I contacted every one of their organizers. Within a couple weeks we had over 200 registrations, and people were emailing us about hotels and hotel accommodations. It was a big wow moment for us."
As for those who are involved in the community and want to do more, Matthew has a few tips.
“If you want to teach training, my advice is not to focus on the curriculum. Everywhere I go, everyone wants me to submit my curriculum. I’ve taught a lot of classes, and I’ve learned that every group of students is going to be different. I can’t tell you how many times I had a curriculum and thought I was going to go through the whole thing, and instead I wind up talking about something else. Make sure you understand your students, and teach them what they want to learn about.
“And it’s not even all that difficult,” he added. “Here’s what I do for Global Training Days. You know what happens when you make an event that’s free anything? You get all sorts of people registering. So I keep the classes as small as possible — so whenever anybody registers for the event, I don’t give them the address unless they give me their phone number. When someone registers, I call them and ask them, 'what do you want to get from this training,' 'what is your experience,' — and I can’t tell you how many people have thanked me for that. Requiring a phone call with the registration helped keep the classes smaller, and limit it to people who were actually new to Drupal and would be in proper context of attending a GTD. So, that’s my advice — know your audience. Don’t assume everyone knows about Drupal. Don’t worry about a set curriculum — go with what you students are actually looking to gain from the experience."
For those who are looking for more tips, tricks, and knowledge from Matthew, his wisdom will soon be available at your fingertips in paperback form.
“I’m working with Packt publishing on a collection of ‘Drupal 8 Administration Cookbook Recipes.’ They’re step-by-step tutorials that teach admins to do everything from getting started with Drupal to doing advanced site building stuff, and it's coming out in later this summer. I’m pretty excited about it! I’ve always read a lot of the Drupal books and I’m excited to make a contribution. I started on my own but have had a Drupal business for several years, and have had several people who have worked for me. The book is literally just a collection of all the same questions every client always asks. It’s almost like a tell-all: here’s step-by-step documentation for how to do everything. Now stop asking us,” he added with a laugh.Front page news: Planet Drupal
Read our Roadmap to understand how this work falls into priorities set by the Drupal Association with direction and collaboration from the Board and community.
The team is back from New Orleans and thankful for the time we had to spend with the community, attending sessions, presenting sessions of our own, and sprinting with you throughout the Con. As individuals, we’re all members of the community, and as an organization we're proud to hold the home of the community in trust.
Because of DrupalCon North America, May is always a busy month for the Association engineering team. We're preparing our sessions, ensuring that the testbots will be running smoothly for DrupalCon sprints, and polishing new features and ideas to share with the community. Here's what's new:Drupal.org updates Composer repositories moving towards stable
At the end of April, we launched the Alpha of our Composer façade, providing Composer repository endpoints on Drupal.org for Drupal 8 and Drupal 7. At DrupalCon New Orleans, we gave a presentation on the architecture of the Composer façade, and our plans for next steps. We also received some great feedback from users who helped us test the alpha release, and in May we've focused on moving Composer from an alpha release to a more stable environment suitable for use on production Drupal sites. We'll be following up soon with a more detailed blog post about Composer, when that more stable release is available.
If you want to help test the Composer service, you can learn more about Drupal.org's Composer repositories.New documentation content types
As previewed in our session at DrupalCon New Orleans, we're modernizing Drupal documentation with two new content types: Guides, and Documentation Pages. Documentation Pages will be organized in Guides, which will be curated by maintainers. We're also bringing a new visual design to documentation, re-organizing documentation by major version of Drupal, and developing a call-outs feature to help highlight key information like best practices or important changes in minor versions.
In May, we made an initial deployment of these content types to Drupal.org, though access is presently restricted to administrators while we work with the Documentation Working Group to sort out our initial migration plan. In June, we hope to deploy a migration tool, allowing users to convert existing documentation Book Pages and their children into the new Guides and Documentation Pages.CKEditor
We've also deployed CKEditor to Drupal.org. The WYSIWYG editor is now available on the Section, Page, and Post content types, as well as the incoming Documentation Guide and Documentation Page types. CKEditor brings a more robust editorial experience to Drupal.org, and as it gets wider use we’ll expand it to additional content types. We also want to allow time for the Dreditor maintainers to update to support the change. As a long-term goal, we hope that some of the features of Dreditor may be reimplemented as CKEditor plugins and directly available to every Drupal.org user without the use of a 3rd party browser extension.Sustaining support and maintenance DrupalCon Dublin full site launched
At DrupalCon New Orleans, we launched the full site for DrupalCon Dublin. The call for papers is open now, as is registration, so submit your sessions and purchase your tickets soon. DrupalCon New Orleans had the most sessions submissions ever for a DrupalCon, and the standard of quality was incredibly high. We're hoping that DrupalCon Dublin will see just as many wonderful submissions.DrupalCon Baltimore announced!
As is tradition, we also revealed the location of the next DrupalCon North America. In 2017, DrupalCon will be in Baltimore! At the closing session, the engineering team launched the splash page for the upcoming event, with travel information, hotels, and important dates.
And if your organization would like to sponsor DrupalCon Baltimore, you can find more information and our prospectus on the site as well.Infrastructure
We made several tweaks to Drupal.org infrastructure in May as well. We updated the Git Twisted daemon, which serves as the backend for the Drupal.org Git repositories and packaging process. We rebuilt our staging infrastructure at OSU/OSL. And finally, with the generous support of new Technology Supporting Partner OpsGenie, we updated our internal pager rotation for infrastructure alerts.
As always, we’d like to say thanks to all the volunteers who work with us, and to the Drupal Association Supporters, who made it possible for us to work on these projects.
We also want to say a special thanks to the departing leadership team at the Drupal Association: our former executive director, Holly Ross, who is moving on after building an incredible team and a great culture throughout the entire organization; Matt Tsugawa, our CFO; and Josh Mitchell, who has lead and mentored the engineering team.
Megan Sanicki, our former COO, is taking on the mantle of Executive Director and we're looking forward to where her leadership will take us