How to Inject Custom Logic to Intercept and Modify API Requests and Responses Without Touching Frontend/Backend Code

When working with APIs, dealing with raw JSON for input and output can be cumbersome, especially when there’s a convenient UI designed specifically for that API. However, the UI doesn’t always keep up with changes in the service, and sometimes it’s useful to emulate changes in the service along with the UI to finalize decisions on what and how to modify.

Such a solution could allow you to replace actual API responses with tailored fake data, eliminating the need for altering real data or making code changes. It would accelerate slow requests by recording and replaying them, significantly reducing wait times for UI tweaks. Developers could simulate edge cases, such as handling 4XX or 5XX status codes or testing long strings, by modifying response bodies, status codes, or headers, or add a debug/verbose mode automatically to all calls to an API. Additionally, it would enable frontend development against unimplemented APIs by mocking responses, facilitating progress even when backend endpoints are incomplete. For integration with third-party platforms that only function on production domains, such a solution could reroute requests, allowing safe testing on production without risk. Furthermore, it would support debugging by adding artificial delays and simulating network errors, ensuring robust handling of edge cases. In essence, mocking servers on production in your browser would streamline development, enhance testing, and ensure a more resilient application.

For example, in the case of SAP Commerce Cloud, there’s the OCC API for product search. The frontend sends a request to /occ/v2/<website>/products/search, and the results are displayed as facets, a list of found products, and pagination. Suppose you’re considering a service that preprocesses the search query: if the user’s input matches a certain pattern, additional conditions and keywords need to be added to the API request. How to mock such service without touching the frontend and backend code? Agree, it would be very convenient to build a quick prototype in the same UI, test it, and then decide whether to build a full system, modifying the backend/frontend code accordingly.

In this article, I will explain how to model such a change on a real website in just half an hour. All you need is access to the website through a browser and — optionally — Python with a few libraries installed. The result will be a working prototype of the future solution. You can manually test it with typical queries, discuss the change with product owners, and demonstrate it in the browser.

As the “victim,” I chose the SAP Commerce demo store, specifically its search functionality. The goal is to inject my custom code between its backend and frontend, and as a demonstration of capabilities, automatically apply certain facets for specific queries. As a result, for the query “cheap cameras,” I will actually see cheap cameras and not tripods, for example.

 

Now, let’s move on to the technical side of the topic.

The task essentially involves intercepting the request from the frontend to the backend, modifying the request content, and, if necessary, altering the response.

The simplest solution for this is a browser extension that performs all the work.

Let’s start with the extension-only option (option 2).

I am aware of the following browser extensions that help with this task:

Requestly (requestly.io). Overall, it works great, but modifying the body payload is only available in the paid version.
Netify. It is supposed to work well but the Scripting feature is in beta and, as of May 2024, has some issues. For modifying JSON payload and response, the scripting feature is the must.
ModHeader and ModResponse. You can replace the response JSON but you can’t process the payload and response with your own code.

Likely there are also a few similar extensions less popular than listed above. It seems that the browser built-in functionality available for such extensions is not sufficient to build a good product, and involving the special API service (like Requestly’s) creates additional complexities in terms of unclear security and the need to pay for such a service. Therefore, from the options presented, only Netify seemed suitable for me, as it doesn’t send anything anywhere and free. However, unfortunately, it is not fully developed and scripting is not supported fully.

To address this problem, it was simpler for me not to go through various browser-only solutions and study their limitations and possibilities, but instead to involve Python into this small project with a script that fits on one screen. Moreover, the solution with Python turns out definitely more flexible. However, it at least requires installing Python on the computer.

For a demo, I used a Spartacus electronics demo store. This instance is not under my control, so both frontend and backend are kind of black boxes for me.

Let’s attempt to search “cheap cameras” there:

We see that the results are not relevant.

Just for the demo purposes, let’s imagine that the right way of handling the “cheap cameras” query is setting up a price constraint to $50-$199.99 and “category” to “Digital Compacts”. In fact, it is probably not the best solution, but it is good enough for our prototype and to demonstrate the idea.

At the very end of this article you will find a simple script of a proxy that receives all calls on https://127.0.0.1:443/<path>?<query> and redirects them to https://spartacus-demo.eastus.cloudapp.azure.com:8443/<path>?<query>. The redirection is synchronous, the response from the remote API is delivered to a caller as it were a direct call to the API.

the browser extension, Netify, helped me to redirect all browser calls to the localhost

the rule should be active:

Let’s add some sample logic to query transformation (see the full code on https://github.com/raliev/proxy-for-apis-demo):

def modify_query_string(query_string, args):
    # change query string if needed
    #EXAMPLE - EXTREMELY SIMPLIFIED FOR DEMO PURPOSES
    params = args.to_dict()
    query = params.get('query', '')  
    if 'cheap' in query and 'camera' in query:
        if ':price:' not in query and ':category:' not in query:
            query += ':relevance:price:$50-$199.99:category:576'    
    params['query'] = query
    new_query_string = '&'.join([f'{key}={value}' for key, value in params.items()])
    return new_query_string;

Seamless API Transition Testing with Proxy Interception

This approach can be used for testing services in development with a frontend that doesn’t yet support them. It’s particularly useful when legacy and new services have similar or even identical API contracts, allowing developers to see if the site will function with the new service by hypothetically using it in place of the legacy one. Typically, such changes require updating the frontend configuration (usually fixing the URL). However, a proxy solution often allows developers to quickly get feedback from the system on whether the new service will work as a replacement for the old one without making immediate changes to the frontend configuration.

By intercepting and modifying API requests and responses through a proxy, developers can simulate how the frontend interacts with the new service. This can accelerate the testing process and highlight potential issues early, enabling more efficient debugging and development. It also allows for a seamless transition and evaluation of the new service’s compatibility, providing valuable insights before making any permanent changes to the frontend codebase.

Activating debug mode for APIs

In the proxy code, you can enable debug mode if the API supports request-scoped verbose responses. A typical example is the parameter debug=true in the URL or a debug attribute in the POST/PUT request payload which triggers verbose output in the JSON response. It’s convenient that this mode can be activated even in the production version, allowing you to see extended information in the API response.

Simulating Bugfix Before it is Applied to the Backend/Frontend Code

For example, if a bug is discovered in the backend or frontend code that blocks testing or studying the system’s behavior on real data, normally you would have to wait for the bug to be fixed. However, there is often a workaround, such as removing the field in the request that causes the error or correcting the value of this field to a value that does not cause the error but does not affect the functionality being tested. This task can also be solved through the proxy.

Request and Response Logging with Proxy

This proxy can also be used for logging all requests, headers, and payloads of both the requests and responses from the service. This is particularly useful when the frontend makes numerous requests during a session, making it inconvenient to study this activity using the developer console in the browser. Especially it is important when you need to catch hard-to-reproduce one particular case. By centralizing and logging all interactions through the proxy, you can easily analyze the data flow, identify issues, and gain insights into the system’s behavior without the clutter and limitations of the browser’s developer tools. You can also log data conditionally to catch that rare situation when a service responds with something specific for example, you want to log all requests causing the erroneous response which occurs intermittently.

The Code

Also available on https://github.com/raliev/proxy-for-apis-demo 

from flask import Flask, request, Response
import requests
from flask_cors import CORS
import json
#create certificates to support https
#openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes
#the URL to which all incoming requests will be redirected to
TARGET_URL = 'https://spartacus-demo.eastus.cloudapp.azure.com:8443'
def modify_target_path(path):
    # change path if needed
    return path;
def modify_query_string(query_string, args):
    # change query params if needed
    return query_string;    
def modify_headers(headers):
    # change headers if needed
    return headers;    
def modify_json_payload(json_payload):
    return json_payload;    
def modify_response_body(response_body):
    return response_body;
app = Flask(__name__)
CORS(app)  # Enable CORS for all routes
@app.route('/', defaults={'path': ''}, methods=['GET', 'POST', 'PUT', 'DELETE', 'PATCH', 'OPTIONS'])
@app.route('/<path:path>', methods=['GET', 'POST', 'PUT', 'DELETE', 'PATCH', 'OPTIONS'])
def proxy(path):
    url = TARGET_URL + "/" + modify_target_path(path)
    query_string = modify_query_string(request.query_string.decode('utf-8'), request.args)
    if query_string:
        url = f"{url}{query_string}"
    print("NEW URL:" + url)
    headers = modify_headers({key: value for key, value in request.headers if key != 'Host'})
    # Handle CORS preflight requests
    if request.method == 'OPTIONS':
        return Response(headers={'Access-Control-Allow-Origin': '*',
                                 'Access-Control-Allow-Methods': 'GET, POST, PUT, DELETE, PATCH, OPTIONS',
                                 'Access-Control-Allow-Headers': 'Content-Type, Authorization, x-anonymous-consents'})
    json_str = request.get_data().decode('utf-8')
    json_obj = {}
    if json_str:
        try:
            json_obj = modify_json_payload(json.loads(json_str))
        except json.JSONDecodeError:
            pass
    bodypayload = json.dumps(json_obj)
    if request.method == 'GET':
        resp = requests.get(url, headers=headers)
    elif request.method == 'POST':
        resp = requests.post(url, headers=headers, data=body)
    elif request.method == 'PUT':
        resp = requests.put(url, headers=headers, data=body)
    elif request.method == 'DELETE':
        resp = requests.delete(url, headers=headers, data=body)
    elif request.method == 'PATCH':
        resp = requests.patch(url, headers=headers, data=body)
    response = Response(modify_response_body(resp.content), status=resp.status_code, headers=dict(resp.headers))
    response.headers['Access-Control-Allow-Origin'] = '*'
    response.headers['Access-Control-Allow-Methods'] = 'GET, POST, PUT, DELETE, PATCH, OPTIONS'
    response.headers['Access-Control-Allow-Headers'] = 'Content-Type, Authorization, x-anonymous-consents'
    return response
if __name__ == '__main__':
    #openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes
    app.run(port=443, ssl_context=('cert.pem', 'key.pem'))