Continuous Monitoring

Continuous Monitoring: What Is It, And How Does It Impact DevOps Today?

Written By: Daniel Kulvicki, Solutions Director at Calavista


DevOps has made it possible for organizations to develop and release stable applications faster than ever. However, an organization with a proper DevOps pipeline should always include Continuous Monitoring through the development lifecycle. Continuous Monitoring (CM) is a fully automated process that provides real-time data in all stages of an organization’s DevOps pipeline. This data interprets any security risks or compliance issues before the application gets to the production environment.

This article will explain what Continuous Monitoring is and how it impacts DevOps today. Let us dive in!

What is Continuous Monitoring?

Continuous monitoring is one of the most critical processes of DevOps. It is an automated process that allows software development organizations to observe and detect security threats and compliance issues throughout the development lifecycle. Continuous Monitoring also provides automated metric reporting to measure the application’s performance and track the user experience trends.

Continuous Monitoring is crucial to all the stages of software development. It enhances smooth collaboration between the development team, Quality Assurance, and the business functionality teams.

For example, the DevOps team releases an application update; the customer service team will depend on Continuous Monitoring (CM) to identify any complaints from the end-user. The development team will automatically address these complaints quickly. Without Continuous Monitoring processes in place, an organization is usually blind to negative customer sentiment.

In simpler terms, Continuous Monitoring provides feedback on errors, security threats, and compliance issues so that the teams can address or rectify these issues faster.

What is the importance of Continuous Monitoring in DevOps?

Continuous Monitoring delivers the visibility needed in order to help drive greater quality for the entire product. Organizations are now using DevOps to develop multiple applications simultaneously. This means that the developers upload their specific code to a central repository consistently. Continuous Monitoring minimizes the chances of incorrect code getting past the various testing environments. CM automatically detects and reports these errors so that the response team can address these issues in real-time.

Other than identifying and reporting errors, Continuous Monitoring comes with additional benefits:

Enhanced visibility and security

DevOps teams rely on automated processes to analyze data across all the stages of an organization’s DevOps pipeline. Continuous Integration and Continuous Delivery (CI/CD) are some of the most crucial steps of DevOps. However, these processes involve consistent changes to the code. Continuous Monitoring ensures that erroneous code does not get to the production environment. It will detect these errors as soon as the developers integrate the code in the central code repository. The response team provides a solution to these errors as soon as they are detected through real-time reporting.

Continuous Monitoring allows the operations team to analyze data throughout the DevOps pipeline. This way, they can track any security threats and address them immediately. CM also ensures that the team does not miss any crucial incidents or trends.

Instant Feedback

Continuous Monitoring involves a constant feedback loop. This feedback is essential to optimizing applications to meet the end-user needs. At the same time, the seniors at an organization can use this feedback to make informed decisions that align with the business goals. DevOps is about delivering rapidly without compromising the quality and functionality of the applications.

Real-Time Metrics Reporting

In a development setting, the teams work together to release multiple apps at the same time. However, without a proper continuous monitoring strategy, this can often pose a challenge. It is due to the rapid and frequent changes from different developers and the combined processes of DevOps methodology. It all needs to happen in a controlled environment with real-time reporting of metrics.

Continuous Monitoring tools provide automated reporting of metrics at each stage of the DevOps pipeline. You will need a tool that can look at the team’s productivity. It is also crucial to have a tool that can analyze your processes’ vulnerability and compliance issues.

Continuous Monitoring alerts the operator in any case of a broken code before the downtime occurs. In some cases, the operator can assign automated actions based on the organization’s risk analysis and DevOps strategy.

Enhanced Business Performance

Executives in an organization can use data from the continuous monitoring processes to make time-efficient and cost-effective decisions. In addition, the business functions team can use the metric report to optimize the sales and marketing processes which will enhance the overall business performance.

For instance, the team can use the data to define the key performance indicators of the business. The organization can also benefit from continuous monitoring and produce a customized DevOps pipeline.

Better Automation

Automation is the backbone of DevOps processes, especially when it comes to metrics reporting. Automation is a necessity for all the stages of DevOps. Now, it becomes even more efficient when an organization integrates deployment automation with monitoring tools.

Not only does this provide better reporting, but it also enhances smooth collaboration between the developers and the operators. They do not need to go back and forth to analyze data and fix issues. Continuous monitoring automation alerts the operators whenever there is a bug in the development phase. The operations team will alert the response team and have the bugs fixed in real-time. This process reduces the chance for bugs to reach the production environment.

With automation, the team can also assign automated actions for repetitive tasks to allow a smooth feedback loop in all development phases. As a result, organizations adopt DevOps to allow faster and continuous delivery of high-quality applications.

Three Types of Continuous Monitoring in DevOps

Security threats and compliance issues are some of the challenges that software development organizations face today. However, a strategic continuous monitoring process allows DevOps teams to foresee these problems. In addition, continuous monitoring helps organizations stop malicious attacks from outside, unauthorized access, or control failures. There are three different areas, or types, of Continuous Monitoring in DevOps that help organizations combat the security threats and compliance issues they’re faced with.

Infrastructure Monitoring

Good infrastructure enhances your application delivery. DevOps teams can use infrastructure monitoring to collect and analyze data to point out any disruptions or incidents that may occur. It includes monitoring the operating system, storage, user permissions, and the overall server status and health.

Network Monitoring

On the other hand, network monitoring looks at the performance, including server bandwidth, latency, and availability. As a result, the operations and QA teams can scale the organization’s resources and distribute the workloads evenly through continuous network monitoring.

Application Monitoring

Lastly, application monitoring analyzes and fixes performance issues. The team can rely on application monitoring to analyze app error rate, uptime, user experience, and system response.

Best Practices for Continuous Monitoring in DevOps

Continuous monitoring should be applied in all areas of the DevOps pipeline for accurate metrics and timely response. Below are the 4 best practices for continuous monitoring:

1. Define the organization's scope of Continuous Monitoring implementation

Like all the processes of DevOps, you will need to identify your scope for Continuous Monitoring implementation. This involves a thorough risk analysis to determine the processes that you will prioritize when implementing CM. For instance, if you are in the finance industry, you may want to analyze the security risks before settling on the processes to monitor.

To do this, you will need to collect as much information as possible about your DevOps Pipeline. Then, by analyzing this data, you can understand what the organization requires to perform at an optimal level.

Choose to monitor processes that will provide crucial feedback that will help you improve your environment to enhance your overall business performance.

2. Use metrics and historical data to determine trends in risk management

Analyzing historical data is an excellent way to decide what to monitor based on risk analysis. For instance, historical data reveals the security threats or compliance issues the company has faced in the past. This way, you can use the trends and apply continuous monitoring to the relevant processes accordingly.

3. Incorporate automation in all stages of developments

Once you identify the processes you want to automate, it is crucial to automate the monitoring process. Automating continuous monitoring leaves the team to focus on other essential tasks. Besides, it aids in risk mitigation as the operators are notified of any security threats that occur. The operator will then alert the response team to resolve these issues immediately.

As with automation, it is best to include continuous monitoring in all stages of the DevOps workflow.

4. Choose an appropriate monitoring tool

Getting the correct DevOps monitoring tool is crucial to successful and consistent tracking. Using the data collected, you can choose a monitoring tool that best suits your DevOps workflow. You should therefore outline your preferred functionalities for your monitoring tool.

An excellent monitoring tool should include reporting and diagnostic features. It should also have an easy-to-use dashboard, one that stakeholders, developers, and operations teams can learn quickly. Continuous monitoring is all about providing relevant data to help improve the DevOps workflow of an organization. Thus, your chosen tool should collect vast amounts of data. It should also include notifications to alert the admin immediately to a security risk, or compliance issue is arising throughout the DevOps pipeline.

Some companies prefer custom-built DevOps monitoring tools, while others will use third-party tools. However, the tools must align with the goals of the organization. In addition, companies should incorporate continuous monitoring in all stages of DevOps as identifying issues arising is crucial to fast and high-quality application delivery.

Conclusion

Now that you have a good idea on what Continuous Monitoring is and the benefits that it grants, our next blog will dive into the actual metrics that allow us to fully gauge the quality of the products we are delivering. If you have any detailed questions about Continuous Monitoring, I would be happy to answer them!


big data fast data machine learning

Big Data, Fast Data, and Machine Learning

Written By: Steve Zagieboylo, Senior Architect at Calavista


While it may seem I’m just trying to work in as many buzzwords as I can, in fact, there really is an important intersection of these three elements. I’ve been interested in both big data and fast data for several years, and my newest tech interest is machine learning. As I have learned about the latter, I see that there are problems that require all three to be truly effective. One application for which I’m looking at bringing together these technologies is in Recommender Systems for brick and mortar shops.

Big Data + Machine Learning ---> Recommender Systems

Probably the first big win for machine learning was Recommender Systems. You’re probably familiar with these in your online shopping, movie watching, or music selection activities, where the website suggests (usually pretty accurately) additional products, movies, or music that you would enjoy. There are a few algorithms for generating these recommendations, but one aspect of them all is that they need a lot of data to “train” the system.

Screen Shot 2018-03-21 at 2.23.25 PM

In a shopping recommender, data is fed into the system as "features." These are measurable pieces of data that the developer thought would be relevant to the buyer biting on additional lures and adding more items to their cart. The developer doesn't have to know how those features are relevant; as long as he is feeding into the machine learning system a reasonable set of features, it will figure out which are significant and to what degree. The developer "trains" the system by feeding in known data with known results. The machine learning program tweaks all the parameters – how much it cares about each feature – until the known data tends to create the known results.

The synergy for big data and machine learning, therefore, is straightforward. There is a ton of data that is potentially relevant, and the developer will need a lot of known examples with all that data available before he can train the system. Since he isn't sure what data will make good features, he has to collect everything, even though some of it will turn out to be inconsequential. It can be surprising which pieces of information actually matter, but he won't find out until actually training the system with all that data in place.

Fast Data

Screen Shot 2018-03-21 at 2.23.37 PM

Fast data systems also deal with a lot of data, but do it in real time. Typically, there is less variety of data types than what you would see in big data, but the quantity can be staggering, and the decisions it makes need to be timely.

Fast Data in Recommender Systems?

Returning to our example of Recommender Systems, there doesn't seem to be much need for fast data. Collecting the feature data for a single user to process through the trained system is not demanding enough to need fast data.

But now, let’s think about brick and mortar shops. Say you’d like to improve the up-sell opportunities to those customers, but to do it in a way that is not as annoying as the ubiquitous “would you like fries with that?” At first, it seems that you know very little about the customer. You don’t have a history like an online store maintains for its logged-in customers, and, unless your salespeople are extremely observant, you don’t have any information about what the customer looked at and considered when making a purchase.

Screen Shot 2018-03-21 at 2.23.44 PM

Or do you?

Mobile phones today broadcast their GPS locations to all who would like to listen (or, more accurately, to all who would like to pay for the “anonymous” data). This data is accurate enough to know not just that the person is in your store, but where in the store he is. If you’re tracking this information, you can know how long the phone spent in each department, even how long was spent right in front of a particular product display. If you could feed this sort of information into a recommender system of the same sort your online store has, very likely you could prompt your salesperson to point out some specific items in the clearance section, or to offer a specific coupon.

However, the data does not come into the system in such easily digestible form, with locations and time spent, the sort of data you could feed to your recommender system. If the customer has connected to your free wi-fi, he is very likely sending a simple location 30 times a minute. When you multiply by all the customers in all your stores, it is a flood of data that is difficult to use for any sort of purchase recommendations. Your big data system could store all this, and could probably crunch it to be useful, maybe tonight, but that’s much too late to do any more than send the customer an email which will probably be ignored. If you can't act while the customer is standing there, you've lost the opportunity.

Screen Shot 2018-03-21 at 2.23.54 PM

The piece that is missing is a fast data. A fast data system can process that stream of GPS locations and build useful information that becomes features in your machine-learning-based recommender system. I don’t know how each datum translates to sales, but that’s the beauty of machine learning – we don’t have to know. If you can generate data that is somehow – probably – relevant, and some known results with which to train the system, then it will do the rest.  As you get more data and more results, both positive and negative, then with each retraining your recommender system will get better and better.

All three pieces are necessary for this complete system: You need the fast data processor to generate the data while it is still relevant. You need a big data system to hold all the inputs over time, what was recommended, and whether or not it succeeded in generating additional sales. And you need a recommender system based on machine learning to be able to improve your recommendations in the future.


application microservices

Breaking an Application into Microservices

Written By: Steve Zagieboylo, Senior Architect at Calavista


I recently started a new greenfield project, where the decision was to use a microservices-based architecture. The application was pretty well defined, including most of the data model, since there was a working prototype, so my biggest first concern was how to break it up appropriately into microservices. I’ve participated in this process before, including making some of the mistakes I’ll try to warn you of, so I developed a way to think about the question that helps with the process.

Why Microservices?

There is plenty of documentation out there telling us why a microservices-based architecture is preferable for a large project, so I’ll just hit the highlights.

  • Clean, well-defined interfaces make for better code. Sure, you could keep the interfaces this clean in a monolith, maybe if you were the only developer. But you can’t when there are tens or hundreds of developers trying to sneak around them, intentionally or otherwise.
  • Separately deployable services allow for quicker turnaround of features and bug fixes. Honestly, I’ve never gotten this far in an application, including this one we just developed. In the early stages, there are still interdependencies, where one service needs certain info from another, so somebody has to create the API, somebody has to make the code that calls it, and it probably the same person doing both. But I truly believe that in a mature version of this product, those sorts of changes will diminish, and most small features will remain internal to a single service.
  • Horizontal scaling becomes more efficient. I have seen this already. Some services have so little load that two instances is plenty, whereas some services scale up to a dozen instances. (We don’t drop below two so that a single failure doesn’t shut us down.)

 

How to Approach the Process

There are no hard and fast rules for how to break up your application into microservices, but these are the things that I’ve found are the key considerations.

Data Model

I tend to be a data-first architect, so, of course, this is where I start. Build a basic Entity-Relationship Diagram (ERD) of your data. This will naturally create some groupings, but it’s the rare data model that isn’t completely connected by relationships. Draw a circle around the major groups. This task is generally obvious, so the rest of this section will focus on how to resolve trickier cases.

Consider whether they are really Joins, or just References

Frequently, you’ll find that you have relationships which are many-to-one, where there is really no reason to connect them at a level through which you can perform a join. For instance, say you have a table of CELL_TOWER, with location, power, date_installed, etc. There is also a table of CELL_PHONE, with a separate table that connects the two. (Even though we assume a phone is only connected to one tower, you put this in a separate table because it changes so fast, and it might be an in-memory table.) You also have a table of TOWER_MAINTENANCE_RECORD.

The TOWER_MAINTENANCE table clearly is in a different microservice from the one CELL_PHONE is in. But which of these two services is CELL_TOWER in?

The answer lies in considering how you’re going to use this data, and where the real join is. Which is more likely: That you’re going to be presenting a list of towers, perhaps filtered or sorted by something to do with its maintenance records? Or that you’re going to present a list of towers filtered or sorted by something to do with the cell phones connected to it? Clearly, the first is far more likely. You might want to get a snapshot of all the phones currently attached to a particular tower, but you don’t need a join to do that. This operation is probably a drill-down from some list of towers, so you already have the tower information in your hand, including its ID. You just need to go to whatever service has the table that connects phones and towers and query against that. This query does want a join to CELL_PHONE, because you’re about to present a list of phones, probably with several columns. The query doesn’t pull anything from CELL_TOWER, though; it just needs the particular tower’s ID to filter against.

Rule of Thumb: When you’re going to be presenting a list of things with filtering and sorting, that’s one important place where you need a real join. The table with the items themselves must join to the tables that have the filtering and sorting parameters, so they have to be in the same microservice.

UI Interaction:

Ideally, any given screen of UI Interaction should interact with only one or two microservices. (Yes, I know we don’t have independent screens anymore, but you know what I mean.) For example, say you have one wrapper screen with everything about a PATIENT. Within it are tabs with different information about the patient: one with a list of VISITs and a drill-down on VISIT; another with a BILLING_AND_PAYMENTs; etc. Almost certainly, those two tabs are accessing two different microservices, and the wrapper with basic patient information (name, date of birth, …) is possibly a third. However, this does not violate my rule of thumb, because you can think of the different tabs as different “screens,” given the somewhat expanded definition of “screen.” In any case, as a rule of thumb, it’s pretty weak as it gets violated frequently. Its value is that it might tip the balance when you are unsure which service should hold a particular table, consider what the UI is interacting with.

The latest trend in microservices is to create Microservice UI, where the UI components that interact with a microservice live in that service and are only included as components in the larger framework. If, for instance, another column is added to one table in one service, then the UI to present and to edit that data is guaranteed to be in the same service. This targeted UI is included as a component, perhaps in several places in the application, but none of those places need to change. I admit that I’ve not used this approach in a real application, yet, but I’ve experimented with it. My experience was that I still needed to make changes in the wrapper to account for changes in the size and geography of the component. However, my experiment was not developed with a reactive UI, and most of the issues I saw would not have been problems if it were.

Rule of Thumb: Try to break up your microservices such that any given “screen” of the UI is interacting with no more than two services. When you do find yourself violating this rule, do so knowingly and have a good reason.

Rule of Thumb: Any UI component should interact with only a single service. Consider putting that UI code in the same repo as the back end code for that service.

Don't Overdo It

The most common mistake I’ve seen (and done myself) is to go overboard in making lots of nanoservices. (A term I thought I had just made up, but a quick Google search shows that I’m not the first one.) Just because this one table of PATIENT_COMMENTS is only accessed on a screen dedicated CRUD of those comments, and there are no real joins to it because it is just a drill-down from a Patient page doesn’t mean that needs its own microservice. Creating a microservice adds some overhead in mental load for understanding the overall application, that somewhat offsets the mental load of isolating a section cleanly. The net result should be to decrease the mental load, not increase it.


Test Driven Development

Test Driven Development

Written By: Jeremy Miller


In the DevOps Methodology Explained blog by Daniel Kulvicki, he introduced the notion of Continuous Testing and Test Automation as part of the overall DevOps methodology. This week, I am going to re-use one of my earlier blogs to kick-off a deeper dive into testing with a focus on Test Driven Development (TDD) and Behavior Driven Development (BDD).

Test Driven Development and Behavior Driven Development as software techniques have both been around for years, but confusion still abounds in the software industry. In the case of TDD, there’s also been widespread backlash from the very beginning. In this new series of blog posts I want to dive into what both TDD and BDD are, how they’re different (and you may say they aren’t), how we use these techniques on Calavista projects, and some thoughts about making their usage be more successful. Along the way, I’ll also talk about some other complementary “double D's” in software development like Domain Driven Development (DDD) and Responsibility Driven Development.

Test Driven Development

Test Driven Development (TDD) is a development practice where developers author code by first describing the intended functionality in small, automated tests, then writing the necessary code to make that test pass. TDD came out of the  Extreme Programming (XP) process and movement in the late 90’s and early 00’s that sought to maximize rapid feedback mechanisms in the software development process.

As I hinted at in the introduction, the usage and effectiveness of Test Driven Development is extremely controversial. With just a bit of googling you’ll find both passionate advocates and equally passionate detractors. While I will not dispute that some folks will have had negative experiences or impressions of TDD, I still recommend using TDD. Moreover, we use TDD as a standard practice on our Calavista client engagements and I do as well in my personal open source development work.

As many folks have noted over the years, the word “Test” might be an unfortunate term because TDD at heart is a software design technique (BDD was partially a way to adjust the terminology and goals of the earlier TDD to focus more on the underlying goals by moving away from the word “Test”). I would urge you to approach TDD as a way to write better code and also as a way to continue to make your code better over time through refactoring (as I’ll discuss below).

Succeeding in software development is often a matter of having effective feedback mechanisms to let the team know what is and is not working. When used effectively, TDD can be very beneficial inside of a team’s larger software process first as a very rapid feedback cycle. Using TDD, developers continuously flow between testing and coding and get constant feedback about how their code is behaving as they work. It’s always valuable to start any task with the end in mind, and a TDD workflow makes a developer think about what successful completion of any coding task is before they implement that code.

Done well with adequately fine-grained tests, TDD can drastically reduce the amount of time developers have to spend debugging code. So yes, it can be time consuming to write all those unit tests but spending a lot of time hunting around in a debugger trying to troubleshoot code defects is pretty time consuming as well. In my experience, I’ve been better off writing unit tests against individual bits of a complex feature first before trying to troubleshoot problems in the entire subsystem.

Secondly, TDD is not efficient or effective without the type of code modularity that is also frequently helpful for code maintainability in general. Because of that, TDD is a forcing function to make developers focus and think through the modularity of their code upfront. Code that is modular provides developers more opportunities to constantly shift between writing focused unit tests and the code necessary to make those new tests pass. Code that isn’t modular will be very evident to a developer because it causes significant friction in their TDD workflow. At a bare minimum, adopting TDD should at least spur developers to closely consider decoupling business logic, rules, and workflow from infrastructural concerns like databases or web servers that are intrinsically harder to work with in automated unit tests. More on this in a later post on Domain Driven Development.

Lastly, when combined with the process of refactoring, TDD allows developers to incrementally evolve their code and learn as they go by creating a safety net of quickly running tests that preserve the intended functionality. This is important, because it’s just not always obvious upfront what the best way is to code a feature. Even if you really could code a feature with a perfect structure the first time through, there’s inevitably going to be some kind of requirements change or performance need that sooner or later will force you to change the structure of that “perfect” code.

Even if you do know the “perfect” way to structure the code, maybe you decide to use a simpler, but less performant way to code a feature in order to deliver that all important Minimum Viable Product (MVP) release. In the longer term, you may need to change your system’s original, simple internals to increase the performance and scalability. Having used TDD upfront, you might be able to do that optimization work with much less risk of introducing regression defects when backed up by the kind of fine-grained automated test coverage that TDD leaves behind. Moreover, the emphasis that TDD forces you to have on code modularity may also be beneficial in code optimization by allowing you to focus on discrete parts of the code.

Too much, or the wrong sort of modularity can of course be a complete disaster for performance, so don’t think that I’m trying to say that modularity is any kind of silver bullet.

As a design technique, TDD is mostly focused on fine grained details of the code and is complementary to other software design tools or techniques. By no means would TDD ever be the only software design technique or tool you’d use on a non-trivial software project. I’ve written a great deal about designing with and for testability over the years myself, but if you’re interested in learning more about strategies for designing testable code, I highly recommend Jim Shore’s "Testing Without Mocks" paper for a good start.

To clear up a common misconception, TDD is a continuous workflow, meaning that developers would be constantly switching between writing a single or just a few tests and writing the “real” code. TDD does not — or at least should not — mean that you have to specify all possible tests first, then write all the code. Combined with refactoring, TDD should help developers learn about and think through the code as they’re writing code.

So now let’s talk about the problems with TDD and the barriers that keep many developers and development teams from adopting or succeeding with TDD:
 

1. There can be a steep learning curve. Unit testing tools aren’t particularly hard to learn, but developers must be very mindful about how their code is going to be structured and organized to really make TDD work.

2. TDD requires a fair amount of discipline in your moment-to-moment approach, and it’s very easy to lose that under schedule pressure — and developers are pretty much always under some sort of schedule pressure.

3. The requirement for modularity in code can be problematic for some otherwise effective developers who aren’t used to coding in a series of discrete steps.

4. A common trap for development teams is writing the unit tests in such a way that the tests are tightly coupled to the implementation of the code. Unit testing that relies too heavily on mock objects is a common culprit behind this problem. In this all-too-common case, you’ll hear developers complain that the tests break too easily when they try to change the code. In that case, the tests are possibly doing more harm than good. The follow up post on BDD will try to address this issue.

5. Some development technologies or languages aren’t conducive to a TDD workflow. I purposely choose programming tools, libraries, and techniques with TDD usage in mind, but we rarely have complete control over our development environment.

You might ask, what about test coverage metrics? I’m personally not that concerned about test coverage numbers, don’t have any magic number you need to hit, and I think it’s very subjective anyway based on what kind of technology or code you’re writing anyway. My main thought about test coverage metrics are only somewhat informative in that the metrics can only tell you when you may have problems, but can never tell you that the actual test coverage is effective in any way. That being said, it’s relatively easy with the current development tooling to collect and publish test coverage metrics in your Continuous Integration builds, so there’s no reason not to track code coverage. In the end I think it’s more important for the development team to internalize the discipline to have effective test coverage on each and every push to source control than it is to have some kind of automated watchdog yelling at them. Lastly, as with all metrics, test coverage numbers are useless if the development team is knowingly gaming the test coverage numbers with worthless tests.

Does TDD have to be practiced in its pure “test first” form? Is it really any better than just writing the tests later? I wouldn’t say that you absolutely have to always do pure TDD. I frequently rough in code first, then when I have a clear idea of what I’m going to do, write the tests immediately after. The issue with a “test after” approach is that the test coverage is rarely as good as you’d get from a test-first approach, and you don’t get as much of the design benefits of TDD. Without some thought about how code is going to be tested upfront, my experience over the years is that you’ll often see much less modularity and worse code structure. For teams new to TDD I’d advise trying to work “pure” test first for a while, and then start to relax that standard later.

At the end of this, do I still believe in TDD after years of using it and years of development community backlash? I do, yes. My experience has been that code written in a TDD style is generally better structured and the codebase is more likely to be maintainable over time. I’ve also used TDD long enough to be well past the admittedly rough learning curve.

My personal approach has changed quite a bit over the years of course, with the biggest change being much more reliance on intermediate level integration tests and deemphasizing mock or stub objects, but that’s a longer conversation.

In my next post, I’ll finally talk about Behavior Driven Development, how it’s an evolution and I think a complement to TDD, and how we’ve been able to use BDD successfully at Calavista.


Minimum Viable Product Brainstorming

I Have a "Killer Idea" And Now I Need Software...

Written By: Andrew Fruhling, Chief Operating Officer at Calavista


Every day, a million and one thoughts fly around in our heads. Sometimes, they're killer ideas — the ones we think will make us never have to work again for the rest of our lives. Other times, they are just products of a very overactive imagination.

However, we know that it is one thing to dream and another to transform that idea into an actual business. Often, a vital part of this transformation is creating a Minimum Viable Product (MVP). This is a term Eric Ries made popular with his book, "The Lean Startup." He defines an MVP as “that version of a new product a team uses to collect the maximum amount of validated learning about customers with the least effort.

In simple terms, an MVP is a product with just enough features to attract customers to validate the product idea as soon as possible. MVPs primarily help companies determine how well their product resonates with the target market before committing a larger budget to it. If you have a potentially killer idea, you should start by creating an MVP to validate the idea.

Approaches to Building an MVP

There are several approaches to building a minimum viable product and this is where things often get challenging – especially for people who do not have much software development experience. I have been in the software development industry for many years, and I have built teams using many different approaches. Some work better than others. People often ask, “How do I get my initial software built?” I thought it would be good to put my thoughts down in writing.

Let’s start with a disclaimer. I have been a customer of Calavista’s twice before joining the company in 2020. As you would expect, I am a fan of the model used by Calavista and I have seen it work well for many customer projects. With that said, I hope this following analysis provides a good overview on why I think this model works well.

For this analysis, I want to make some assumptions. The scope of an MVP is usually constant – hence the term, minimum viable product. That leaves you with three primary levers to consider: Time, Quality, and Cost, plus of course, the overall Risk of the approach. Below, we explore the various approaches to building an MVP and what they would mean for your start-up. Each approach will be scored on a scale of 1 to 5 across Cost, Time, Quality, and Risk (where 5 is the best possible) and an Overall score will be assigned based on an average of the Cost, Time, Quality, and Risk score.

1. Build it Yourself (Overall score: 2.5/5.0)

There are a lot of low-code and no-code approaches to building simple MVP applications. These can work in some cases until your business complexities require a more sophisticated system but expect to have a significant rewrite at some point. I have personally recommended this type of approach to companies when cost is truly the most important criterion. This allows you to minimize your investment while you validate the idea.

a. Cost (5): This has the lowest possible cost as you are building it yourself. Many people take this route if they cannot afford to hire other people to do the work.

b. Time (1): There is often a significant time span and learning curve required to experiment with the capabilities of tools selected. You are often trading costs for time.

c. Quality (2): This element depends on you. If you're an expert developer, you would probably have an MVP with decent quality. However, for most people, this is like watching a DIY home improvement program where the homeowner does not even realize the impact on the quality until a professional shows them what should have been considered.

d. Risk (2): There is quite a lot of risk to this approach, as you often sacrifice quality and time because you want to cut costs.

 

2. Build your own Software Development Team (Overall score 2.0/5.0)

You could also decide to hire developers and build a team to work on your MVP. It is common for a startup to want to build their own team after all if you think you have a killer idea involving software, you think you want to have your own team to build it. Just like building a house, there are many skillsets required to build a software product – even an MVP. Building an initial MVP requires more than just a couple of developers. You will need expertise around architecture, requirements, testing, development tools & processes, and much more.

a. Cost (1): This has very high upfront costs. You typically need to hire multiple people with different skills to build a quality MVP. In today’s job market, these resources can be expensive and hard to find.

b. Time (1): It takes a lot longer than most people think to find good resources, identify the development tools and processes, and build the actual MVP.

c. Quality (4): You will likely get a quality MVP if you build the right team and processes first.

d. Risk (2): Your MVP’s success rests on your ability to hire and retain a team with the appropriate skill set, knowledge, and attitudes.

 

3. Outsource with Contract Coders (Overall score 1.8/5.0)

I see this often and have noticed that while it seems like an attractive approach, it usually does not end well. You can find coders online and pay them for the scope of your project. This may work well for small, straightforward projects that have a clear start and finish. Often, it grows to be multiple independent contractors who do not work together very well.

a. Cost (3): This is relatively low as you don't have to pay the overhead costs for hiring or management plus you can scale up/down as needed.

b. Time (2): It is usually swift to start, but you could soon begin to face issues with developers understanding the project’s scope, defining a holistic architecture, and delivering on time.

c. Quality (1): Despite good developers, the overall quality is often inferior due to interdependencies. In addition, a lack of focus on the overall architecture usually impacts quality.

d. Risk (1): This can be very risky as it might cost you time and quality in the long run as you end up with a disjointed product. Many times this leads to significant re-writes to improve performance, stability, maintainability, and user experience.

 

4. Outsource with Offshore Team (Overall score 2.8/5.0)

Many people talk about how you can outsource your development to companies in countries overseas such as India, China, Mexico, Belarus, etc. There are many cost-effective options but there are also many challenges.

a. Cost (5): Offshore development companies in India and China can provide development capacity at a meager price. On the other hand, locations like Eastern Europe and Central/South America are typically pricier but may offer better overall results.

b. Time (3): Depending on the processes of the company you outsourced your development to, the timing might be shorter. Often offshore companies can scale up a team much faster than you can hire, and the team will have recommendations for tools and processes. On the other hand, some offshore companies tend to have a “never-ending project” syndrome where the final deliverable seems to slip repeatedly.

c. Quality (2): This is where challenges often occur. Quality implies the product works as designed and as intended for the MVP.  It is difficult to ensure quality remotely, especially if you are not familiar with quality development practices.

d. Risk (1): Selecting the right offshore contracting company is difficult and could be costly if you make the wrong choice. There are hundreds (possibly thousands) of options. They all claim to have industry leading best practices and the experience to make you successful. There is often miscommunication around what the MVP needs and its delivery. Between language, culture, and time zone differences, miscommunication is common, especially when you do not have experience working with offshore teams.

5. Outsource with Managed Offshore Team (Overall score 4.3/5.0)

A Managed Offshore Team means you have senior leadership in the US who provide the technical expertise for the project. They typically also offer the practical knowledge to best leverage offshore resources. The onshore management team will include the following:

  • A senior development leader who would be like a VP of Development for most companies
  • A senior architect who provides the technical expertise across the project

Based on the size of the project, these may be fractional resources. This means you’re getting both senior US-based leadership with offshore development costs. If done well, it could deliver the best of both worlds.

NOTE: Many offshore teams will have a US-based resource assigned as part of your project. In my experience, these are typically not the same as the “Managed Offshore” resources described here. Typically, offshore teams assign a US-based account management role to the project rather than an industry veteran with more than 20 years of experience running projects.

a. Cost (3): While not the lowest cost option, you can save a considerable amount on staffing by having a blended team.

b. Time (5): With this approach, you essentially hire a VP of Development who brings a development team ready for your project.

c. Quality (4): The quality usually depends on the repeatable and automated processes you have established. With a great process and collaboration, you often get the best quality.

d. Risk (5): A strong seasoned leadership team with repeatable and often automated best practices that leverages strong offshore development teams with cost-effective rates can significantly reduce your project risks.

 

Final Thoughts

While there are several approaches to creating an MVP, you must carefully choose the one that best suits you. There is not a single best answer for all cases, and you will need to determine which is best for you. The table below summarizes the scores for each approach:
MVP Table

If cost is your primary driver, you may want to consider a ‘Build It Yourself’ option or a good ‘Outsource with Offshore Company’ option. However, if you want to mitigate your risk, I recommend the ‘Outsource with Managed Offshore’ model that provides the best of both worlds.

At Calavista, we have been providing complete, managed teams that are specifically configured to address a customer’s needs and follow the ‘Outsource with Managed Offshore” model described above. Every engagement is led by a Solutions Director (SD) – professionals who have 20+ years of development experience and specific, demonstrated expertise in managing distributed teams. We use a hybrid, Hyper-Agile® development methodology that we’ve refined over the last two decades. These factors enable us to deliver projects with greater than 94% success rate – 3x the industry average. If you would like to talk about how this could work for you, please let us know!


DevOps

DevOps Methodology Explained: Why is DevOps Right For Your Organization?

Written By: Daniel Kulvicki, Solutions Director at Calavista


In the last decade, we have seen significant shifts in software development operations. One of these shifts is the evolution of DevOps, which came to play in 2008/9. Even as organizations continue to adopt the practice, DevOps is still considered an extra when it needs to be a fundamental. In this article, we are going to explore DevOps and why every organization should adopt it.

  • What is DevOps?
  • How does DevOps work?
  • Calavista Tenets of DevOps

What is DevOps?

For years, developers and most system operations teams worked separately. If you have ever worked in a software development company, you would know that these departments don't always agree. And this can always lead to fatal disagreements that delay the development and productivity of an organization.

DevOps is a term used to describe the collaborative effort between developers and the IT operations team to improve the application development and release process. DevOps is derived from ‘Developers’ and ‘Operations.’ It involves agile development methodology and Enterprise Service Management (ESM).

Agile development is a software development approach that focuses on collaboration between cross-functional teams to enhance rapid releases. This approach aims to help the development teams keep up with the fast-evolving market dynamics in software development. Like DevOps, changes in the agile development processes are incorporated continuously to produce a usable version of the application faster without compromising the quality of the output.

DevOps uses agile strategies, including collaboration and automation. However, it also includes the operations team who manage the application in production.

ESM applies IT system management such as system monitoring, automation, and configuration to improve performance, efficiency, and service delivery. This is the practice that brings the operations team to DevOps.

The developers would produce the code and hand it over to the operations team in the traditional software development process. The IT operators would generate or build the application from the code then proceed to test and production. In case of errors or when a client requests changes, the application would go back to the developers, and the cycle would go on.

DevOps changed all this through various practices, including Continuous Integration and Continuous Delivery (CI/CD). Continuous Integration allows developers to submit their code into a shared repository several times a day and throughout development. It is a cost-effective way to identify bugs via automation tools. This increases efficiency since the developers will fix the bugs at the earliest chance. However, it is essential to note that CI does not get rid of the bugs. Instead, it is a principle that aids developers in identifying bugs so they can fix them in a timely manner.

Continuous Delivery makes app or software deployment a lot more predictable. It ensures that the code remains in a deployable state even as developers work to introduce new features or make configuration changes. As with CI, Continuous Delivery enhances frequent app releases with limited instability and security issues. Thus, Continuous Delivery increases not only efficiency but also the quality of the application/software.

CI/CD go hand in hand. They allow developers to make code changes frequently and release them to the operations and quality assurance teams. Together, Continuous Integration and Continuous Delivery increase the rate of production and the quality of applications.

In DevOps, the team works together from development to deployment. This enables organizations to work on multiple projects simultaneously to produce high-quality applications and software.

How does DevOps work?

Developers and system operators differ on a lot of things. But, at the same time, the two teams must work together to successfully deliver software development changes to the end-user (developers write code, and operations gets the code to the end-user). Remember, customer satisfaction is the backbone of any software development organization and this is where DevOps comes in. While it's not easy, bringing both teams together will ease the development and rollout processes.

Let me explain how DevOps works with an example.

If we look at a typical software development team working on an application. The team includes software developers, software testers, DevOps engineer(s), and some other roles like scrum master and business analysts. The software developers write new code and make changes to existing code. The software testers ensure the code works as designed – often, this will include automated tests. The DevOps engineers make sure everything comes together and automates as much as possible. The DevOps engineers typically stand up the development tools and environments, implement a process to “build” the code that typically includes the automated testing, and provide metrics on the development process and the application. The primary goal is to produce an app that is stable and secure efficiently, and the DevOps engineers are a critical part of this team.

Software development can be challenging, especially because clients demand changes all the time. However, DevOps teams implement these changes faster through constant collaboration, communication, and automation.

At Calavista, we like to break DevOps into 6 different tenets. This helps identify various areas of focus for our clients.

Calavista Tenets of DevOps

If your company is about to adopt DevOps or simply looking for ways to improve your current development processes, you will need a solid strategy. It will involve bringing cross-functional teams together, which also means changing the work culture. So, how do you go about it? This section highlights the fundamental principles of DevOps.

When we talk about DevOps, we like to break it down into 6 key areas crucial to the success of your company’s development - deployment processes.

  1. Collaboration
  2. Automation
  3. Continuous Integration
  4. Continuous Testing / Test Automation
  5. Continuous Delivery / Continuous Deployment
  6. Continuous Monitoring

As with other many changes, adopting DevOps will require you to develop a repeatable set of steps for the team. What goes first, and what comes second? Your DevOps team needs chronological steps so they can work together.

For instance, in a typical situation, a simple DevOps pipeline would look like this:

Step 1: Developers write the code
Step 2: Both engineers and IT operators compile the code and check for errors
Step 3: The operation team enables testing and quality assurance to validate and verify the code
Step 4: Deployment - The code is moved to a staging and/or production environment

However, different organizations will have other DevOps pipelines. Therefore, it is essential to define these functions for DevOps methodology to succeed. This must also include a brief description of the automation tools you will use to develop and deploy an application.

Defining your DevOps pipeline allows smooth collaboration between the teams through the production cycle.

Collaboration

DevOps is built on the principles of collaboration between developers, testers, and system operators. The relationship between these teams will determine your production efficiency. Furthermore, the collaboration goes beyond this core team into your business stakeholders and others to ensure the team is properly building what is needed by the end users following an agile methodology. Helping everyone work together effectively will help you deliver great products.

This means that you might have to change your company culture. DevOps will not work for your organization if your developers and IT team don’t work collaboratively. Remember that it is a strategy that involves constant changes that must be validated in real-time.

DevOps is not only a developers-IT team affair. The stakeholders and management must also join the team so that everyone is on the same page. It is beneficial to both the organization and the team at large.

Good collaboration across the development and operations team and the broader organization is crucial to delivering outstanding software products.

Automation

Successful DevOps implementation relies heavily on automation. Therefore, it is essential to use the right test automation frameworks established in the right tool to automate extensive development and deployment pipeline sections. Automation is a combination of tools and scripting to implement a cohesive and efficient development environment and can include any part of the development, testing, and operations activities, including onboarding.

So, what is Automation in DevOps? It is the use of advanced tools to perform tasks that require minimal human intervention. However, automation does not get rid of the human functions in DevOps. Instead, it enhances the entire DevOps pipeline to allow quick releases. We will outline the benefits of automation in DevOps. But, first, let’s define the DevOps process that can be automated.

DevOps Processes to Automate

Ideally, you can automate all the processes of DevOps, but you usually do not have the time to automate everything. Your automation infrastructure will vary from that of another company based on your specific requirements. When thinking about automation, we recommend prioritizing the following processes:

  • CI/CD
  • Testing
  • Continuous Monitoring

DevOps automation begins with code generation. Next, multiple developers will submit their code into the source code repository, requiring Continuous Integration (CI). At this stage, your automation tool should detect bugs and any inconsistencies in the code. This makes it easy for developers to fix the bugs as soon as the system identifies them. Automation also enhances Continuous Delivery (CD) by allowing frequent configuration of new features and other changes to the code. As a result, it is easier to keep the code in a deployable condition at all stages.

Accurate testing is crucial to software development. Automation tools run testing automatically and as frequently as the developers check the code into the repository. With automation, code testing runs throughout the software development cycles. Therefore, it is less likely for a company to release unstable or erroneous applications using automation.

We will look at these processes in the following sections of this article. All in all, automation is one of the crucial principles of DevOps. Below are some of the benefits of automation in DevOps:

  • Reliability of the app
  • Accuracy during code generation
  • Enhances collaboration between the teams
  • Automation also reduces the cost of production by reducing the number of staff needed for development, testing, and deployment
  • It improves the overall quality of the app

Continuous Integration

Continuous Integration enables developers to submit and merge their individually written/modified code into the shared main code branch. For instance, once the product road map is laid out, code generation starts. In most cases, developers will begin with the most critical parts of the source code. Therefore, Continuous Integration requires that individual developers submit and merge their code into a shared repository several times a day.

These changes go through automated testing to identify bugs. Developers will fix any detected bugs as soon as possible to keep the code ready for the main codebase. This allows for smooth workflow and consistency through regular adjustments to the code to meet the set validation criteria. Continuous Integration is also an essential step towards Continuous Delivery, a process that we shall focus on later.

Why is CI so significant in DevOps?

DevOps is based on a set of strategies that fuel software development processes for speedy and incremental release. Continuous Integration allows multiple developers to write and merge their code into a shared codebase. This builds transparency and accuracy throughout the development lifecycle. It ensures that everyone is on the same page during the code generation stage of software development. It also promotes collaboration between the involved departments in an organization.

Through automated testing, Continuous Integration enhances the quality of the end-product. Errors and bugs are identified and fixed early enough before checking in the code into the main codebase.

Continuous Integration starts with building the most critical code to layout the product roadmap. This is followed by automated tests on the code in the shared repository before merging the code in the main codebase. Everyone on the team must update their code multiple times a day. Remember that CI is all about keeping the code in a deployable state at all times. Upon testing, developers should focus on fixing any bugs in the code as soon as they are detected.

Continuous Integration takes us to the following critical principle of DevOps; Continuous Testing / Test Automation.

Continuous Testing / Test Automation

Earlier, we talked about how critical automation is as a component of DevOps. Automation starts immediately after the developers start writing the code and run throughout the development lifecycle. Continuous Testing goes side by side with Continuous Integration. It involves continuously reviewing the minor changes integrated into the codebase to enhance the quality of the product.

Continuous Testing is an excellent way to attain a frequent feedback loop on the business risks at all stages of development. First, developers merge the code, and the quality assurance team takes over through test automation. Unlike traditional software development methodologies, the QA team doesn’t wait for developers to write all their code. For many companies, test cases are actually written before the code, in what is called Test Driven-Development (TDD) and the test cases will simply fail until the code is written. In all cases, testing needs to happen as soon as the code gets to the shared repository. At this stage, bugs and errors are detected, and developers can fix them immediately.

Continuous Testing puts customer satisfaction in the mind of the developers at the idea stage of development. The quality assurance team uses Test Automation to check for software stability, performance, and security threats when developers integrate changes into the standard repository. Thus, Continuous Testing enhances the quality of the application and speeds up the development process.

An organization will need to develop a Test Automation strategy when laying down a software development roadmap. It can include unit testing, UI testing, load testing, API Integration testing, among others. Test Automation plans vary from one organization to another depending on the DevOps pipeline and selected metrics.

Continuous Testing and Test Automation bridges the gap between Continuous Integration (CI) and Continuous Delivery (CD). Upon testing and fixing detected bugs, the operations team proceeds to Continuous Delivery - A process that ensures that the code remains in a deployable state throughout the development lifecycle.

Continuous Delivery / Continuous Deployment

Continuous Delivery is the logical next step after Continuous Integration and Continuous Testing and is an integral part of almost every Calavista project. Automating the delivery mechanism massively reduces delivery issues, delivery effort, and delivery time. However, please note that even though the delivery is automated, manual release mechanisms may still be in place for moving a release from one environment to another – especially customer acceptance and production environments. Continuous Deployment automates even these delivery steps. This process goes hand in hand with Continuous Integration. Minor changes are integrated into the central code repository in a deployable state throughout the development lifecycle upon testing the code.

In other terms, Continuous Delivery (CD) is a combination of the processes we have discussed, i.e., building the code, testing, and merging the code in the main codebase in short cycles. In this step, the software is only a push-button away from deployment (release to end-user). At this phase, the team reviews the code changes and verifies/validates the code changes to ensure that the application is ready for release. When the criteria is met, the code is pushed to a production environment.

The changes incorporated in the CI/CD phase are released to the customer in the Continuous Deployment phase. The code changes go through Test Automation to check for quality and stability before releasing them to the production environment. Continuous Deployment focuses on customer or End-user satisfaction. For instance, if a user makes a bug report, developers can make changes to the code. The changes will be automatically deployed upon passing an automated test. The deployment will fail if the newly written code does not pass the test. Therefore, Continuous Deployment reduces the inaccuracy from recently applied code and thereby maintains the software's stability. Continuous Deployment is the process that enables developers to add new features or updates to a live application.

Continuous Deployment checks the code for the stability of newly integrated code changes before releasing the software to the client or end-user via Test Automation.

Continuous Monitoring

Continuous Monitoring (CM) is usually the last phase of the DevOps pipeline. However, it is just as important as any other phase. The continuous model of operations means that code changes occur rapidly. As a result, Continuous Monitoring helps the DevOps team with the proper insight into what and how their system is operating.

So how does CM work?

DevOps engineers are required to help other teams support and maintain their application. Continuous Monitoring is put into place to enable support to be proactive instead of reactive. Metrics and KPIs are introduced for Continuous Monitoring to enable more visibility and insight into how the production code is both developed and running. In addition to metrics, centralized logging is usually put into place to expedite the diagnosis of issues. These tools bring together a way to monitor all aspects of an application in support of creating a better product.

Continuous Monitoring reduces app downtime. It is because the team is aware of the app's performance as well as threats on a continuous basis. Besides, bugs are detected and fixed in real-time, thereby enhancing customer satisfaction.

The primary goal of Continuous Monitoring is to ensure maximum performance of the app. This can only be achieved through responding to customer feedback, monitoring user behavior, and deploying changes as often as needed.

Conclusion

I hope this blog has helped you gain a better understanding of DevOps and how we break it out at Calavista. Hopefully on your next project you can reference our Tenets and see how better you can fit DevOps into your organization. Please feel free to reach out as we always love to talk DevOps!


Calavista Joins The DesignSpark Innovation Center

Written By: Daniel Kulvicki, Solutions Director at Calavista


Big news!!! We wanted to share with everyone our amazing new Bryan - College Station location at the DesignSpark Innovation Center. The Innovation Center is the perfect place for inspiring entrepreneurs and long-standing Aggie businesses to coalesce, to further grow the startup ecosystem near the Texas A&M campus. Calavista has always loved to bridge the gap from startup idea to product and we are very excited for the attention focusing on the startup community here. The Aggie graduates at Calavista feel privileged to help the A&M community in any way possible and this is a great opportunity.
 

The DesignSpark Innovation Center Photo Tour

Below are some shots of outside the Innovation Center at Lake Walk in Bryan, Texas. What a beautiful building and landscape!
 

DesignSpark Innovation Center
Courtyard at the DesignSpark Innovation Center

 

Exterior of the DesignSpark Innovation Center
Photo By: The DesignSpark Innovation Center

 

DesignSpark Innovation Center Exterior
Photo By: The DesignSpark Innovation Center

 

In the image below, the building on the right is a great gym with an incredible, professional-quality basketball court. This gym is open to members of the Innovation Center and surrounding businesses. It's a great place to workout and build camaraderie within the area.

 

DesignSpark Innovation Center Gym
DesignSpark Innovation Center gym exterior.

 

Check out the Stella Hotel that is right across the street. The Stella Hotel is a great place to stay and have events near the Innovation Center. The tower over the lake offers great views to anyone who climbs the stairs. We also like to hang out at the POV Coffee House next door to the Stella Hotel.

 

Stella Hotel near the DesignSpark Innovation Center
View of the Stella Hotel near the DesignSpark Innovation Center.

 

Now to the inside. First up is the Calavista office at the  Innovation Center. We are so proud, and excited to be right in the middle of the building! We are moving in later this month and will be ready for visitors very soon!

 

Calavista office at the DesignSpark Innovation Center
Calavista office at The DesignSpark Innovation Center

 

Below is one example of the artwork in the building. There are great settings throughout the building. Each time I look at this wall, I find another interesting and inspiring quote.

 

Quote wall at the DesignSpark Innovation Center
Wall of quotes at The DesignSpark Innovation Center.

 

Here is a good look at the inside of the building. You can see the artwork wall on the left, and our office is just past the shuffleboard table that you see on the right.

 

Interior of the DesignSpark Innovation Center
Photo by: The DesignSpark Innovation Center

 

Check out the events that occur at the Innovation Center on a regular basis. Look out for a Calavista employee if you come to an event, as we love to attend these!

 

Events at the DesignSpark Innovation Center
Recurring events at the DesignSpark Innovation Center.

 

Although I could share a ton more pictures, we would love for you to come see our office in Bryan for yourself! If you are in, or around Bryan/College Station, feel free to ping me or anyone at Calavista for a meeting at The DesignSpark Innovation Center. We would love to get to know you and tell you how we can help you and your business take the drama out of software development!


Approaches for Generating Realistic Test Data

Written By: Steve Zagieboylo, Senior Architect at Calavista


In this post, I'm going to discuss some approaches to obtain realistic Test Data without compromising the security of any customer's sensitive data. In the world of health care software, this is referred to as Personal Health Information (PHI), but the concept exists in financial software, document management, ... really, everything.

Why is Realistic Test Data Important?

The data you test against must include a lot of completely unrealistic data: the corner cases that you have to make sure you handle correctly, but almost never come up in real life. However, it is equally important to have a good quantity of realistic data, justified by, “You just never know.” I can’t count the times that a special case came up in the real data that we hadn’t considered as a corner case. If your application handles large amounts of real-world data, there’s just no substitute for testing it with large amounts of real-world data.

What Aspects are Important?

  • Quantity. Having too little data will hide a lot of performance problems, such as an N+1 problem in your queries. You don’t need the individual developers’ systems to have the same quantity as a production system, but the primary QA server should.
  • Quality. One of Calavista’s customers has a very cool product that does intelligent analysis of the data produced in a hospital stay. This includes checks, for instance, of the drugs prescribed, considering the age of the patient. If the data were just randomly generated with actual drugs and actual dates, but no consideration of making them ‘realistic’ (i.e., drugs that a doctor would actually have prescribed for a patient with this profile and this diagnosis), then the intelligent analysis code will freak out. Tests created with random data would just be useless.
  • Interconnectedness. The manner and scale in which the different parts of the data model connect should be realistically represented. For example, compare a business in which Customers typically make 3 or 4 orders a year compared to one where they typically make thousands of orders a year. The approaches to both the UI and to the data storage will be quite different.
  • Approaches to Create Realistic Test Data

    There are three general approaches:

  • Generate the data by hand.Someone who is an expert in the system as well as in the specific field would have to create a realistic data set. Even with tools to support the effort, this would be a humungous undertaking and it would still be a challenge to make data "realistic," especially medical data. It would need to include Patients and Visits and Symptoms and Diagnoses that all fit correctly together and have realistic quantities.
  • Use a tool that generates the data.This is industry specific, of course, but often there are tools that can do some of this work. In the case of medical data for instance, there are some tools for generating, from whole cloth, data that adheres to the HL7 standard. There are several good choices, if all you’re looking for is legal HL7 messages, but nothing that can automatically generates a fully coherent set of data
  • Start with real data and obfuscate it. This approach is the only one we have been able to find that really solves the problem. The challenge here is to remove all sensitive information from the data, but still leave it adequately realistic. This approach is the subject of the rest of this paper.
  • Obfuscation / Data Masking

    PHI data is more than just the names and addresses of the patients. It is important that even a determined investigator would not be able to take obfuscated data and reverse the process to figure out whom the data describes. Therefore, not only names but birthdates, dates of visits, locations, even some specific combinations of symptoms might need to be obscured. However, changing the dates too much might change the analysis of the illness: Symptoms of osteoarthritis are much more alarming in a twelve year old compared to the same symptoms in someone who is seventy-three. Proper obfuscation, or data masking, creates characteristically intact, but inauthentic, replicas of personally identifiable data or other highly sensitive data in order to uphold the complexity and unique characteristics of data. In this way, tests performed on properly masked data will yield the same results as they would on the authentic dataset.

    Tool Recommendations

    SOCR Data Sifter
    On open-source tool for obfuscation. It has a setting for exactly how obfuscated the data should be, where the lowest setting is unchanged and the highest setting leaves the data unrecognizable.
    Documentation Link
    Source Code Link
     
    SQL Server Dynamic Data Masking (DDM)
    In SQL Server, as of SQL Server 2016, there is a Dynamic Data Masking feature that allows you to specify data masking rules on the database itself, with no need to change the data. Whether or not a particular user can see the real or the masked data is based on user privileges. Here is the best overview of the feature that I’ve found: Use Dynamic Data Masking to Obfuscate Your Sensitive Data
     
    Roll Your Own
    I am usually a big advocate of buy over build. The software you imagine is always bug-free and does just what you want. It isn’t until you’ve spent a bunch of time working on it that it acquires the bugs and feature compromises that make you unhappy with the existing tools. In this case, however, it is a pretty small bit of code and purely internal, so if it chokes, you haven’t embarrassed yourself in front of customers. The biggest risk is that you have to make sure that no sensitive data is getting past it, which is pretty straightforward to check. The biggest bit is to have a serious review of the entire database schema to make sure all-important fields are being modified.
     
    Approaches:
    You can randomize the data just with SQL, with clever use of the RAND command. Here is a great article on this approach: Obfuscating Your SQL Server Data.
     
    If the code for your data model is cleanly isolated in a library, you can make a new application, pull in that library, and build your tool with that, using the same technology your developers are already comfortable with. If your code is not so cleanly isolated, a riskier approach is to write the obfuscation code inside the same code base, where it is only possible to trigger it with a completely isolated command. I’d recommend putting what amounts to a two-factor authentication to trigger this code; just think what a disaster it would be if it somehow gets built and then run inside the production system. It should only be included if a TOOLS_ONLY flag is set in the build process, and can only be run if the OBFUSCATION_TOOL configuration flag is set in the application configuration. Of course, neither flag should be on by default. Even so, carefully consider the impact that an error – or even a rogue developer – could have on a production system before using this approach.

    This extra code might need other additional controls on it, as well. For instance, if you have tables that are flagged as Insert-only, you will need to change the settings and add new code to your lowest level code to enable your obfuscation code to write the obfuscated versions of the records. Again, these functions should have two levels of enablement, such that they are not even built in the code that goes to production, and if they accidentally do get built, they won’t run because they do nothing if the OBFUSCATION_TOOL configuration flag is not set.

    Steps:
    These steps should be built into the DevOps tools so they can be run easily, or even are automated to run every day. For one Calavista customer, we set up a process where part of the nightly process was to copy all the production data, obfuscate it, and install it in the QA system. It was incredibly helpful in tracking down tricky bugs, because even though the names and dates were different, the IDs of the records that triggered the problems were consistent, and that was all that was saved into logs.

  • Clone the data. This is no harder than just taking a recent backup, standing up a new database instance, and restoring from backup. This MUST be done inside the controlled data space, where only users who are approved to see PHI data have access.
  • Run the obfuscation code on the alternate data set, changing it in place. There are a number of tools for this step, with some recommendations below. Alternatively, you could create these tools yourself.
  • Create a "backup" of the alternate data set, which now has only obfuscated data.  Only this backup can leave the protected zone.
  • Delete the alternate data set.
  • In the non-protected zone where the QA systems live, bring in the alternate backup and restore from backup to the database where you want to have this latest data.
  • Additional Consideration: Logging

    This is more a general thought for any system with sensitive data, but it intersects with this approach. When logging specific record information, such as when you are logging that an unusual process is being invoked for a certain record, the developers should know to be conscious of what data is sensitive, and never to put that data in the log. For most applications, it is acceptable to put naked IDs in the logs, since those do not mean anything to someone without access to the database, anyway. Fortunately, the IDs are exactly what you want to be able to hand to developers who are going to be debugging against the obfuscated QA database. They still match up to the problem records, whereas the names, addresses, etc. do not.

    Conclusion

    When dealing with sensitive data of any sort, including health care data, it is important for QA systems to have realistic data upon which to operate. Having small amounts of hand-generated data is going to allow bugs and problems to creep in, and will make them very difficult to track down with they are found in the production system. There is really no substitute for getting the real production data and having a copy of it, properly obfuscated, in the QA environment. Calavista recommends performing this step automatically, as part of the nightly process. Even though getting this set up represented work that had to go on the schedule, it inevitably more than pays for itself in the long run.


    Challenges of Data Migration

    Written By: Steve Zagieboylo, Senior Architect at Calavista


    In my last blog, I talked about how we estimate new projects, and I included the offhand comment that Data Migration is always harder than you think it will be. The purpose of this blog is to provide a few examples of why I find this to be the case.

    What is Data Migration?

    Data Migration, in the context of web projects, is needed when you are making a major change to the product, such that the data model has significantly changed. Tools like Evolve or Liquibase aren't going to cut it, because those tools are meant for point changes to the application, with small changes to the data model that you could describe in a couple of commands. You need a migration if you are going to be switching your users over to a completely rearchitected version of the application. It’s still the same tool your customers are paying for, but with the major revision you’ve fixed those areas that are clunky, and (hopefully) simplified the abstraction to make lots of new functionality possible.

    However, you still have a ton of data that needs to be transferred to the new model. Your customer wants everything that they already appreciate about your product to remain. Of course, the most important part of that, to them, is their data. If any of their data is lost, it won’t matter how much better the new version of your application is, they will be unhappy with it. Getting the migration right, therefore, is an absolute requirement to keeping your customers happy.

    Practice, Practice, Practice!

    Specific challenges are addressed below, but the first and most important takeaway from this article is that you need to have practiced the migration step dozens of times before you do it for real. If you haven’t done several complete migrations on a true copy of production data, and successfully brought up the new system with the migrated data, then you aren’t ready to try it on production. You need to start with a staging system that is functionally identical to your production system, including versions of peripheral elements and some horizontal scaling of the pieces that scale horizontally. (You can have just three instances instead of nine, but if there are multiple instances in production, you need to test with multiple instances.) You should replicate the entire process on the staging server, all the way from shutting it down (with some active users at the time being politely booted) to switching the DNS entries so that the next users will log on to the new system.

    Testing, Testing, Testing!

    Part of the importance of doing all the practices is the testing that goes along with it. Identify what the key indicators are that represent correct data and automate the comparisons. For instance, if you have Customers who each have Orders, you should be able to get the count of Orders for each Customer, sorted by customer name on both systems, and the arrays should match exactly. It’s no good to migrate all your data if it introduces inaccuracies in the data when you do it.

    Challenges

    Data Model Mismatch

    There are lots of ways in which the data model can be a mismatch, and the approach to each is very different.

    1. Indirection Layer Added.Perhaps a many-to-one relationship is becoming many-to-many, or references are being bundled so they can be changed together. You'll need code to create the new layer of indirection.

    2. Data Being Collapsed. You had a hack in the old system where you made extra records because it couldn’t account for some more complex entities that your new data model can represent. There are two stumbling blocks here. First, make sure you’re not double-processing your complex records, once for each sub-part you run into. Second, make sure you compensate for the differences in record counts when you do your testing.

    3. Less-used Columns Abstracted. You may have made a number of columns in a table that are only used in certain cases, and you find yourself making more and more of these. Instead, you roll them all into a separate table of named properties.

    4. Special Cases Abstracted. Special cases, such as dependency requirements, which used to exist in the code with hard-coded references are now abstracted in the data. Not only do these need to be captured correctly, but old data that didn’t meet the dependency but was “grandfathered in” now has to be handled. (This is a very real and very painful issue that came up for us recently. Our solution was a spreadsheet import that referenced specific records by ID that had to be modified as they were transferred.)

    5. Old Data no Longer Tracked. In a complete rewrite of a client’s product, there were some old features which were no longer supported, but we did not want to lose the old data associated with them. It would only be referenced by internal users in unusual circumstances, but it shouldn’t disappear. Our solution was to dump it into a text file and keep it as ‘attached documentation,’ a field we had anyway.

     

    Sheer Quantity of Data

    You will need to know how long it will take to migrate the data to know how long you’ll be shutting down theData Migration Anecdote system. Of course, by the time you’re doing it for real, you’ll have practiced the whole process many times, and you’ll know exactly. But before you get there, you might want to perform an experiment or two to have an idea how long it will take. This might surprise you, and it may affect the entire approach. (See the sidebar.)

    If you find that your migration is too slow, there are some tricks you can do.

    • Bigger Transactions. If you are wrapping your rows processed each in a transaction, you are spending a lot of time starting and stopping transactions. Instead, bundle a group of rows into a single transaction. You don’t want to get too large, though, because one error means the whole transaction needs to be rolled back. A bundle of 50 or 100 will mean that the transaction overhead is small compared to the real work.
    • Multi-Process. If you are reading from one database, doing work, then writing to another, all done very serially, there are two ways you can process in parallel that are pretty easy to do. First, if your data is pretty independent, you can break it up according to the key of the source data and kick off several threads, each of which are assigned a block of the source data. Second, and this turns out to be easier than you think, is to read and process in one thread and then write in another. Either way, the goal is to make the database the bottleneck (typically the one you’re writing will max out first), rather than your processes. Be careful, because you don’t want to create too many threads, all eating up memory, when all they are doing is waiting for their chance to write to the database. You really only need two, so that one thread is reading from A while the other is writing to B and vice-versa. If there is a fair bit of processing, perhaps use three threads.

     

    Corrupted Data

    If you’re migrating from a system that has lived and evolved for several years, don’t assume that the data is perfect. It isn’t. Minor errors have crept in, from failed operations that weren’t properly in a transaction that would roll back; from hand-editing the SQL; from requirements changes on the data, where old records are being ignored; from who knows what. Your migration had best not choke and die on some corrupted data. Your first full pass through the data will probably die, and your second, and … That's why we practice the migration several times.

     

    Data Changing While Migration is Occurring

    If you do not have the luxury of shutting down the application while you are migrating the data and moving to the new system, then you must deal with the possibility of data changing after the migration has started. This is not too bad if you have immutable data that is all time-stamped, but if you had thought that hard about your data model before, would you really be migrating off of it? There is no one-size-fits-all answer here, but consider these approaches.

    • Take a backup, restore to another instance, and migrate from that. At least then you’ll have a clean snapshot to work from and a very clear timestamp.
    • Try harder to convince the bosses that access to the system can be shut down from 3 to 6 am on Sunday morning. It’s a horrible weekend for you, but it’s better than data loss.
    • If all data-changing events are done through an event bus, you can replay the feed (probably through its own translation to the new system), but if you had that architecture, I wouldn't have to tell you.
    • A poor man’s version of the last point: Add to the old system code that writes all the changes to a file at the same time it is saving it, such that you can read the file back and replay it. Combined with the first bullet point, this can give you a safe transfer.

     

    DevOps

    In addition to the coding effort for the migration, there’s going to be a big DevOps challenge for coordinating the migration and then the entire changeover from the old production system to the new one. Part of the practice has to include automating the entire process, from creating the database instances, firing up the migration code with the correct accesses to old and new databases, executing it, firing up the new application code, and switching DNS or load balancer to direct new connections to the new code. If any of these are being done manually, make sure there is a checklist, and stick to it religiously in your practices as well as, of course, in the real thing. In the final practices, don't let the DevOps person do any task that isn't on the checklist, no matter how obvious and everyday the task is.
     

    Summary

    When you’re making a major change to an entirely new version of your web application, you are already going to have some users who metathesiophobic (afraid of change -- I had to google it.) so they are already likely to grumble. The last thing you can afford is to have data issues as well. You need to budget lots of extra time to make sure that the data migration goes smoothly, and you need to practice the whole process, end to end, several times before the D-Day when you do it on the production system. Have a well-rehearsed process governed by a checklist that is so complete anyone could follow it. And GOOD LUCK!


    blueprint

    Estimating Software Projects in an Agile World

    Written By: Steve Zagieboylo, Senior Architect at Calavista


    Calavista boasts an impressive 90+% on-time in-budget track record. As I pointed out in an earlier blog, I consider this SEVEN times better than the industry average of 30%. And yet, Calavista is very firmly an agile shop -- we know the folly of waterfall methodology, especially for greenfield projects. It doesn’t really work for real projects that real people care about, because a lot of the decisions you make in designing software can’t really be made without putting it in front of real users. Many people will claim that you can’t really do accurate estimations in an agile environment, because you can’t estimate if you don’t know the final target, but those people are wrong.

    Lawrence Waugh (CEO and Founder of Calavista) likes to compare the process to house construction. Let’s say you have purchased some land, and you want to build a house. You go to an experienced house builder and you say you want a 3-bedroom colonial with 2.5 bathrooms, and you want to know how much it will cost. After looking at the lot and asking you a few questions -- room sizes, dining room, quality of furnishings and finish -- he will give you a number. It will be a suspiciously round number, but once you work out all the details, it will turn out to be pretty darn close. The contractor can do this because he has done it a lot of times before, and he knows the sort of basic costs and the sorts of problems that can crop up, and he makes an allowance for a reasonable number of them. In other words, he has built houses much like the one you want to build, and he knows what the process is going to be.

    Software is similar, though there is a much wider range of targets than in house building. Since starting at Calavista, I’ve worked on a project that was done in 4 months with a team of 5 people, and I’ve worked on a project that was 18 months reaching MVP (and continued from there) and peaked at 35 people. So how do we estimate such a wide variety of projects?

     

    Understand the Goals and the Scope

    When we are making an estimate for a new piece of software, of course, the first step is to understand what it is supposed to accomplish. This is different from “what it is supposed to do.” The latter question leads down a dangerous path which ends in waterfall specs. We are not, yet, trying to document user flow or data models, we are just trying to capture the basics of who the users are and why they will be using the software. Who the users are might be equivalent to roles, -- it certainly should include all the roles -- but might have a few more. The “accomplishment” level of specificity is closer to “Manage the list of items in the store” rather than the specifics of adding/removing items, changing the prices, etc. This is the equivalent of learning that the house is to have 3 bedrooms or 5.

    List Major Areas of Activity

    For each role, list the things that they want to get done using the software. This is a subset of the stories you will end up with, it is just the important ones, the happy path. A complete set of stories includes all the odd cases -- e.g. users wanting to back up and start over in a process -- but I’m not going to get to that level of detail in making an estimate. I know from the dozens (hundreds?) of times that I have created user flows, that these extra stories will always be there, and my experience in creating includes those, so my estimate will, as well. This is the equivalent of learning that there will be 2.5 bathrooms, a large dining room, and a swimming pool.

    Sketch Out the Data Model

    Your mileage may vary on this step, but I continue to be a “Data First” architect -- that’s just how I think. I make a simplified Entity Relationship Diagram (ERD), not a full-blown database diagram, and I don’t imagine that it is 100% accurate, or even 60%, but it leaves me in a state of stress if I don’t at least scribble this down on paper. If I can feel the shape of the data as it fills and flows, that helps me complete the next step.

    List Out All the Screens

    Next I write down all the screens that the app will present. In modern Single Page Apps or hybrids where a lot happens on a single page, this is not as clear as it used to be, but it works just as well to think of it as a multi-page app with relatively little reuse of screen real estate on a page.. The app hasn’t actually been designed, yet, so I’m not really talking about literal screens that will be presented to a user, necessarily. It’s really just the functionality of user interaction, and these pieces might go together very differently but will still end up having all the parts I’m considering.

    A UI-centric architect might do this before the previous step, and be perfectly successful doing it; it’s a matter of preference.

    For each screen, I enter three values, Back end cost (in person-days), Front end cost (also in person-days), and a fudge factor which ranges from 2 to 10. Having lots of fields and a complicated layout increases the front end costs. Having both data to be fetched and data to be saved increases the back end cost, plus the complexity of the data needed and saved increases it. (This is where my data model helps.) I don’t get down to the individual field level, but I try to be aware of ‘tricky’ fields, such as a field that will have to include a type-ahead against data that is filtered by selections in other fields.

    The fudge factor is usually around 4, but these things increase it

    • This part of the data model is not that well understood, so I expect changes.
    • This part of the user interaction is not that well understood.
    • There’s a ‘science project’ involved, such as interaction with some outside library that I’ve never used before.
    • There is workflow. It’s always harder than you think.
    • There is more than one sub-table involved, where list management is needed at multiple levels.  For example, Customers have Orders and Orders have Items Ordered and Items have Features (size, color, quantity, etc.)
    • There is user input we can’t completely control, such as an upload of a spreadsheet for batch processing. Users have an amazing ability to mess these things up in unexpected ways.

    To continue the analogy to house building, we’ve now specified the sizes of the rooms (more or less), the size of the pool, the length of the driveway, and the quality of the appliances. We are not going to get any closer than this in the estimation level. Once we actually start the project, that’s when we’ll do the equivalent of hiring the architect and putting together the completed plans, including all the details -- closets, plumbing, vents, etc.

    Add Non-Interaction Features

    I add these in the same list, but they have a 0 for the Front End cost. These are things like emails that get sent, background tasks that perform data maintenance, batch processing tasks, data migration from an earlier version of the product, etc. Data Migration, by the way, is always way harder than you think, just go ahead and double it, AND add a large fudge factor.

    This is also the step where I think hard about the non-functional requirements, such as performance, security models, time zone issues. If any of these are outside the bounds of the normal stuff I’ve done several times, I’ll add extra tasks to cover them. I don’t mean to imply that they are necessarily small afterthoughts just because they come at the end. Many of these are pretty significant tasks, possibly even being a proportional overlay to the existing work, such as multi-tenancy, which more or less doubles all the back end costs.

    Apply the Formula

    My formula for the actual costs is:

    (base cost) * (1.25 + (fudge factor)/5)

    The result is that each task with a fudge factor of 4 gets approximately doubled, and higher or lower fudge factors are a little more or less than doubled. This is what I’ve found, just from experience, that the time I expect a task to take needs to be doubled to cover actually completing the task, with completed unit tests, responses to code reviews, and getting it merged into the Development branch. The 1.25 is a factor to account for developers spending time in meetings, as well as time lost to vacations, holidays, and sick time.

    Estimate Personnel vs. Time

    Next, we make one or more plans with people and deadlines, such as “4 back end, 3 front end for 8 months,” taking the requests of the customer into consideration. If they are looking to meet a hard deadline, then we figure out the number of front end and back end engineers we will need to get all the work done before the deadline. If they have been less specific, we might put together a few scenarios with different team sizes and different target dates.

    Add the Other Parts

    After that, we add QA, Requirements Analyst(s), UI/UX, DevOps, and the time for Calavista Solution Director and Architect who together will shepherd the project. We usually also include a “Sprint Zero” for kicking off the project, getting source control, Jira, Confluence, Communications, and the rest of the development infrastructure into place. Some of these are proportional to the amount of work, some are fixed costs.

    Make a Proposal

    Finally, we bring this all together into a proposal. We know that if we get the contract, we will have a framework and enough time to create something that will accomplish the customer’s goals. There will be changes, of course, to the different pieces: the pages, the layouts, the secondary goals and tasks, and the delivery; but with this process we have been very successful in delivering software that makes our customers successful, within the time we’ve actually predicted, which is our ultimate goal.