Return to Workshop

Testing Resiliency with Fault Injection

Your application is working great now with the new version of the user profile service. But the previous version caused performance issues, and future updates may cause issues in other areas of the application. How can you test how your application behaves when a failure occurs?

You need a way to simulate failure in the service mesh. By doing so, you can test if your application functions correctly in a degraded state. This concept is more generally known as Chaos Engineering. In Chaos Engineering, you test your software by causing breakdowns in the production environment.

Istio provides ways to inject failure at the application layer with Delay Faults and Abort Faults using virtual services. Let’s try this with the paste board application.

Abort Faults

View the virtual service in your favorite editor or via bash:

cat ./istio-configuration/virtual-service-userprofile-503.yaml

Output (snippet):

...
  http:
  - fault:
      abort:
        httpStatus: 503
        percent: 50
    route:
    - destination:
        host: userprofile
        subset: v3
...

This configuration tells Istio to inject 503 errors into 50% of the traffic sent to the user profile service.

Deploy the routing rule:

oc apply -f ./istio-configuration/virtual-service-userprofile-503.yaml

If you aren't already (from previous labs) - Send load to the user profile service:

while true; do curl -s -o /dev/null $GATEWAY_URL/profile; done

Inspect the change in Kiali.

Navigate to 'Graph' in the left navigation bar.

If you lost the URL, you can retrieve it via:

echo $KIALI_CONSOLE

Switch to the 'Versioned app graph' view and change to 'Last 1m'.

Change the 'No edge labels' dropdown to 'Requests percentage'

Click on the connection between the 'app-ui' (square) and 'userprofile' service (triangle).

On the right, you should see roughly 50% of requests returned as HTTP errors.

Kiali Graph with abort faults

Let’s test the application in the browser.

Navigate to the 'Profile' section in the header.

If you lost the URL, you can retrieve it via:

echo $GATEWAY_URL

Refresh the brower a couple of times.

Sometimes you will see the profile page show up; other times, you will see ‘Unknown User’.

Injecting an abort fault is a great mechanism to test how your application handles failure. In more complex service meshes, you can use this to identify and prevent cascading failures in which an erroring service causes a chain of other services to fail.

Delay Faults

Version 2 of the user profile service had a performance issue. You can synthetically simulate this scenario with delay faults in Istio.

View the virtual service in your favorite editor or via bash:

cat ./istio-configuration/virtual-service-userprofile-delay.yaml

Output (snippet):

...
  http:
  - fault:
      delay:
        fixedDelay: 5s
        percent: 50
    route:
    - destination:
        host: userprofile
        subset: v3
...

This configuration tells Istio to inject a 5 second delay into 50% of the traffic sent to the user profile service.

Deploy the routing rule:

oc apply -f ./istio-configuration/virtual-service-userprofile-delay.yaml

If you aren't already - Send load to the user profile service:

while true; do curl -s -o /dev/null $GATEWAY_URL/profile; done

Let's inspect the change in Kiali's embedded Jaeger view.

From the Graph view right click on the userprofile service (the triangle shape). Then click "show traces"

Graphed Service - Right Click Menu

Notice some the traces are about 5s in duration while others are in the millisecond range.

Injecting a delay fault is a great mechanism to test how your application handles slow outbound service calls.

Traces to User Profile Service with Fault Delays

Clean up

Revert the changes you made before ending this lab.

oc apply -f ./istio-configuration/virtual-service-userprofile-v3.yaml

Summary

Congratulations, you configured fault injection with Istio!

A few key highlights are:

• We can test the resiliency of our service mesh using Delay Faults and Abort Faults
• Abort Faults synthetically inject 50x errors in responses to a service call
• Delay Faults synthetically add latency in responses to a service call
• Jaeger can capture performance delays with distributed tracing

---

Workshop Details

Domain
Workshop
Student ID

Configure Workshop

---

Return to Workshop