My experiences attending to the “community” in an open source company

I recently returned to managing the forum for an open source web application which is used to create custom processes. I have worked on and off for the company that develops this application since 2009, but I have been away from the forum for the last 3 years. Being confronted with the questions and demands of the community on a daily basis gives me a different perspective from most of the other people working in our company.

It is both a burden and a joy to be inundated with technical questions every day. When I login, I see a dozen new posts, all demanding my attention. Most people who post have a gnawing problem they want resolved or a burning question that they have spent the last 20 minutes searching for an answer in our wiki. I take a bit of professional pride in the fact that I often answer the questions on the forum which nobody else can handle, including the people in the support department. After years of answering questions and documenting the software, I have gained a profound understanding of how the application works.

There are some people who ask questions in the forum because they are too lazy to read the documentation. I generally just give them a link to read and try to not waste too much time with them, because answering one question, means they will be back with 5 more in the future, so it is best to teach them to read before they ask more. Most of the time, however, people ask because the documentation is lacking or unclear. It does not help that most of it was written by people who learned English as a second language.

Roughly a 1/4 of the posts on the forum are questions about how to do something, where the only solution is to insert custom code in the process. These are questions like, “How do I assign the next task to the manager for the user who started the case and then route the case back to that same user?”

Another 1/4 of the posts are from programmers who are already hacking together their own code and need help. Those are the most interesting posts, because they often teach me something new. If they are asking, then the answer is usually not obvious, and it often takes a bit of hunting on my part to figure it out. They are also the most appreciative people on the forum, because they understand how hard it is to dig through the source code. Our application has over 400,000 lines of source code and most of it was written by people who did not take the time to comment their code. A couple years ago, the management started insisting that there needed to be documentation for every function. What was written in the comments by the developers is often not any more informative than the names of the function and its parameters. The code is filled with functions that take arrays or objects as parameters, without any documentation about what should be placed in them. The only way to figure out how a function works is to search through all the source code and look at examples of its use to see what gets passed to the function and what comes out of it. I often have to jump through the code in 3 or 4 different files just to figure out what to place in each array or object before passing it to a function.

The interface to our software is very attractive and trite phrases like “user-friendly” and “drag-and-drop” are often employed to describe it. Once you start developing processes with the software, however, you start noticing all these little things that just don’t work right. It is hard for me to judge whether our software is better or worse in this regard than other applications, but I notice the bugs because I see them every day and I am often the one who reports them. Like the sausage maker who notices all the gunk in his own sausage, I can’t use our software without finding the flaws.

During my time at the company, I have filed roughly 1400 bug reports. A couple hundred of those bugs are minor text changes to ungrammatical phrases written by English-challenged developers. Nonetheless, most of my bug reports are either about things that don’t work right or suggestions to improve the software. Most users of the software just accept the limitations and go on, but some people post on the forum asking how to get around this or that annoyance. The most frustrating part of my job is seeing the same problem month after month without any resolution in sight. It is especially annoying for me as a programmer, when I include code in the bug report to solve the problem, yet the developers ignore it or refuse to implement it.

The best part of my job is when I get to write code to work around some limitation in the software. Yesterday, I spent all day writing code to get around all the problems with uploading files. The principal interface for uploading files hasn’t been changed since I started working with the software in 2009. After uploading a file, there is no way to change its filename or change the original comment or add even add an additional comment to the file. There is no way to specify how many files need to be uploaded or specify different file types for each upload. There is no way to add JavaScript or HTML code to customize the interface or add additional fields to the interface.

For these reasons, most users prefer to upload to file fields inside forms, which are highly customizable. However, if they have multiple files to upload, then they have to place those file fields inside the rows of a grid. That way they can add as many rows to the grid as they like and upload any number of files. The problem is that after they submit the form, there is no way to view the files afterwards that were uploaded. They can go to another part of the interface to view all the files that have been uploaded in every case, but they have to have special permissions to access that part of the interface and the files submitted through file fields cannot be organized into folders.

To overcome these problems, I wrote some PHP and JavaScript code to allow users to both upload and download files inside the rows of a grid and edit the comments and filename for those files. I literally had to do half a dozen hacks in my code to get around things that don’t work right in the software. For example, there is no way to dynamically set the URL and text of a link in a grid, so I had to create hidden fields in the grid and then add JavaScript code to set the links. There also is no way to clear a file field, except to use some jQuery trickery. Of course, my code doesn’t work on the mobile app, because the programmers haven’t implemented a way to execute code in the mobile version when a new row is added to the grid. Still, it makes me feel accomplished in my job to know that I am solving real people’s problems which they complain about in the forum.

As I was creating this hack to get around the limitations of uploading files, I got into an argument with my manager about what kind of information we should be publishing on the wiki, which is the official documentation for the software. Anything which smacks of hackery, he wants to relegate to the blog or the forum, where nobody will find it. I look at it and say this problem could go years without being fixed, so we better give people solutions right now. Besides, I don’t want to waste my time creating a solution, when nobody will be able to find it because it isn’t in the official documentation. To me, it is far better to have code examples that might get broken by future changes in the software, than have no code examples in the documentation and leave users frustrated because they see no workaround.

From the point of view of my manager, however, every code example is one more thing to test after every upgrade. I agree that we should be testing our code examples periodically, which is why we need to hire documentation writers who also know how to program so they can test code. What is more important, however, is to have the documentation team watching the forum, because the community is very good at catching the problems in the code examples after they upgrade the software.

What typically happens is someone posts on the forum saying something like,“I can’t clear the selected file in a file field.” I try it and see the same problem. Then, I file a bug report about it or ask the person on the forum to file one. If I have the time, I play with the JavaScript and figure out a way to clear the file field, so I post the solution on the forum. Then, I post that same code on the wiki, because I know that others will have this same problem and they can’t wait months or years until the developers get around to fixing it. If it gets fixed in a future version, then it can be moved to a note saying, “if using version X, then use this code to clear the file field.”

These sorts of things in the documentation make the software look unprofessional to some eyes, but it looks even more unprofessional in my opinion when a user of our software has to present a new process and her manager says, “hey, I can’t clear the file after I mistakenly selected it.” It looks really unprofessional when the user says, “Oh, there’s a bug report on that, but I have no idea when it will get fixed.” Every user of our software that I know would rather be able to cut and paste some custom code from the wiki to fix the problem rather than tell her manager that there is no solution. What is especially annoying for the users is having to hunt through dozens of posts on the forum to find a solution to the problem or asking on the forum and having to wait days for someone to respond with “Oh, there is a post on the forum or blog about that somewhere. Search for it.”

The good thing about code examples is that they automatically weed out most of the people who don’t want to experiment. If they bothered to copy and paste custom code into their process and test it, then they are willing to also take the time to post on the forum if there is a problem. I have read tens of thousands of posts on the forum, and never once has someone said, “I just tried the code from the wiki and it didn’t work, so this software is worthless or I’m never using this software again.” When I interact with people who has discovered a problem with our code examples on the wiki, I usually respond, “Oh, you are right! Thanks for noticing that it doesn’t work in version X or situation Y. I have updated the code example on the wiki.” Everyone feels good about the interaction. The person on the forum feels smart for discovering a problem and helping to fix it and I’m happy to know the code example is being used in the real world.

Unfortunately, I don’t have workarounds for most of people’s problems, so I see lots of posts on the forum about people asking for solutions and being frustrated because some bug has never been fixed and it is holding up deployment of their processes. I have no idea how many people have stopped using the software because of some limitation or bug in the code, but I suspect that it is many more than the number who decide not to use the software because the official documentation has code examples to workaround problems.

Although this post probably sounds like a long diatribe against our software, the developers and the management team, I suspect many software companies which produce customizable open source applications confront these same problems. People in their communities want solutions, yet the management is reluctant to provide code examples that might break from one version to the next and workarounds for limitations in the software.

Part of the problem is the fact that I am the only person in the company who is interacting on a daily basis with the larger community of users who aren’t paid clients. As the manager of the forum, I’m the only person who has to answer the same question over and over, and deal with people who are frustrated by a problem or limitation in the software. If the management of the company spent 10 minutes every day reading people’s posts on the forum, they might decide that fixing bugs and removing limitations in the software is a higher priority than developing new features and special plugins. They also might be more sympathetic to me posting code on the wiki that helps people fix their problems.

More importantly, they wouldn’t see the community of people who use their software as freeloaders and resource drains for their company. The community often finds bugs first and points out the limitations in our software before our paid clients. The community goes out and tests our software in different environments, but our paid clients are only using the two operating systems which we officially support. For years our software wasn’t being tested on more than the one version of PHP, MySQL and Apache found in CentOS 6, so it fell to the community to test it in all of the other distributions with different versions of the LAMP stack.

Part of the reason why our proprietary “enterprise” plugins are so poorly debugged is because we have so few people using them. Code which is used by a handful of paying clients is inevitably going to be less tested than code used by hundreds of people in the community. I have long believed that we should have an automated system where members of our community can sign up to test our enterprise plugins for limited time periods, just to help debug them and make sure that they work properly with each new release of the software.

People in the community also find all the holes in our documentation. Almost every time someone asks a question in the forum about a particular feature in the software, I see that the documentation wasn’t clear on that particular point or didn’t even cover the topic. Most of the code examples in our wiki initially came from a question asked on the forum. After answering the specific question, I adapt the code example for general use and post it on the wiki. Having an active community helps us find our bugs, see the limitations in the software, and improve our documentation, yet I am one of the few people in the company who sees these benefits.

While the company where I work uses the term “open source” to describe itself, open source operates more as a way to market the software and grow its user base, rather than a fundamental part of the DNA in how the company operates. Our code is developed behind closed doors, so no one in the community can download it and test it while in development. We have a large quality assurance team, but it is very bad at finding the bugs, compared to the community which tests our software in ways we can’t anticipate and in many different environments.

We use a proprietary program for source code management, which is only accessible to the development team. As a member of the documentation team, I can’t access the code in development until a private beta version is released inside the company a couple weeks before the formal launch. I have no way to look through the commits to see what is getting changed and what bugs are getting fixed. For a while we did public beta releases of our code, but that practice has been abandoned. After a version of our software in development was shared by our professional services with a client, it was decided to not allow anyone outside development to have access. There is very little testing of the code outside of development before release and the development department is located in another building, isolated from the support, documentation/training and commercial services departments.

There have been attempts within the company to bridge the communication gap with the development department, with weekly bug meetings and using HipChat to plan future development of the software. Nonetheless, the development team has almost no interaction with the larger community of users or their concerns.

Even our bug management isn’t open to the public. We have a public bug site, but development uses a private bug management system which is only open to members of the company. We are told to report all the bugs we find on the private system. Of course, anything reported on the private system has higher priority, so our community is excluded from the system and can’t contribute. Needless to say, fixing bugs reported by the community is not a high priority.

For most of this year, nobody in the company was even reading the forum, so most questions of a technical nature weren’t getting answered. Again, the community simply wasn’t deemed a priority. I kept bringing up the issue with my manager, but responsibility for the forum bounced between the support and documentation departments, with nobody wanting to take charge. In the end, the forum was given back to me to manage. I suspect that I got reassigned to work on the forum because nobody else wanted to deal with it. At the end of the day, the company needs someone to attend to the forum, because it is a way to reassure clients that we will keep answering their hard technical questions, even after they are no longer paying for commercial services and their support contracts aren’t designed to handle coding questions. Management doesn’t want me to spend too much time answering people’s questions, filing bugs for people on the forum, and writing code examples for free, but they do grasp the idea that we present a poor image to the world when we don’t respond to queries in our own forum.

Attending to the forum and the larger community in general is a role that I relish, although it inevitably frustrates me because I don’t have the time to do it properly. I don’t know what happens in other open source companies, but I suspect that the issues I see in our company are not unique. There is an inevitable tension in any company which offers its software as open source to the public, but has an enterprise version with additional functionality for its paying clients. Hopefully, my experiences highlight some of the issues which arise out of this tension and point out some of the potential problems which should be avoided if possible.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s