rest api paper

[ Top ] [ Prev ] [ Next ]

Representational State Transfer (REST)

This chapter introduces and elaborates the Representational State Transfer (REST) architectural style for distributed hypermedia systems, describing the software engineering principles guiding REST and the interaction constraints chosen to retain those principles, while contrasting them to the constraints of other architectural styles. REST is a hybrid style derived from several of the network-based architectural styles described in Chapter 3 and combined with additional constraints that define a uniform connector interface. The software architecture framework of Chapter 1 is used to define the architectural elements of REST and examine sample process, connector, and data views of prototypical architectures.

5.1 Deriving REST

The design rationale behind the Web architecture can be described by an architectural style consisting of the set of constraints applied to elements within the architecture. By examining the impact of each constraint as it is added to the evolving style, we can identify the properties induced by the Web's constraints. Additional constraints can then be applied to form a new architectural style that better reflects the desired properties of a modern Web architecture. This section provides a general overview of REST by walking through the process of deriving it as an architectural style. Later sections will describe in more detail the specific constraints that compose the REST style.

5.1.1 Starting with the Null Style

There are two common perspectives on the process of architectural design, whether it be for buildings or for software. The first is that a designer starts with nothing--a blank slate, whiteboard, or drawing board--and builds-up an architecture from familiar components until it satisfies the needs of the intended system. The second is that a designer starts with the system needs as a whole, without constraints, and then incrementally identifies and applies constraints to elements of the system in order to differentiate the design space and allow the forces that influence system behavior to flow naturally, in harmony with the system. Where the first emphasizes creativity and unbounded vision, the second emphasizes restraint and understanding of the system context. REST has been developed using the latter process. Figures 5-1 through 5-8 depict this graphically in terms of how the applied constraints would differentiate the process view of an architecture as the incremental set of constraints is applied.

The Null style ( Figure 5-1 ) is simply an empty set of constraints. From an architectural perspective, the null style describes a system in which there are no distinguished boundaries between components. It is the starting point for our description of REST.

5.1.2 Client-Server

The first constraints added to our hybrid style are those of the client-server architectural style ( Figure 5-2 ), described in Section 3.4.1 . Separation of concerns is the principle behind the client-server constraints. By separating the user interface concerns from the data storage concerns, we improve the portability of the user interface across multiple platforms and improve scalability by simplifying the server components. Perhaps most significant to the Web, however, is that the separation allows the components to evolve independently, thus supporting the Internet-scale requirement of multiple organizational domains.

5.1.3 Stateless

We next add a constraint to the client-server interaction: communication must be stateless in nature, as in the client-stateless-server (CSS) style of Section 3.4.3 ( Figure 5-3 ), such that each request from client to server must contain all of the information necessary to understand the request, and cannot take advantage of any stored context on the server. Session state is therefore kept entirely on the client.

This constraint induces the properties of visibility, reliability, and scalability. Visibility is improved because a monitoring system does not have to look beyond a single request datum in order to determine the full nature of the request. Reliability is improved because it eases the task of recovering from partial failures [ 133 ]. Scalability is improved because not having to store state between requests allows the server component to quickly free resources, and further simplifies implementation because the server doesn't have to manage resource usage across requests.

Like most architectural choices, the stateless constraint reflects a design trade-off. The disadvantage is that it may decrease network performance by increasing the repetitive data (per-interaction overhead) sent in a series of requests, since that data cannot be left on the server in a shared context. In addition, placing the application state on the client-side reduces the server's control over consistent application behavior, since the application becomes dependent on the correct implementation of semantics across multiple client versions.

5.1.4 Cache

In order to improve network efficiency, we add cache constraints to form the client-cache-stateless-server style of Section 3.4.4 ( Figure 5-4 ). Cache constraints require that the data within a response to a request be implicitly or explicitly labeled as cacheable or non-cacheable. If a response is cacheable, then a client cache is given the right to reuse that response data for later, equivalent requests.

The advantage of adding cache constraints is that they have the potential to partially or completely eliminate some interactions, improving efficiency, scalability, and user-perceived performance by reducing the average latency of a series of interactions. The trade-off, however, is that a cache can decrease reliability if stale data within the cache differs significantly from the data that would have been obtained had the request been sent directly to the server.

The early Web architecture, as portrayed by the diagram in Figure 5-5 [ 11 ], was defined by the client-cache-stateless-server set of constraints. That is, the design rationale presented for the Web architecture prior to 1994 focused on stateless client-server interaction for the exchange of static documents over the Internet. The protocols for communicating interactions had rudimentary support for non-shared caches, but did not constrain the interface to a consistent set of semantics for all resources. Instead, the Web relied on the use of a common client-server implementation library (CERN libwww) to maintain consistency across Web applications.

Developers of Web implementations had already exceeded the early design. In addition to static documents, requests could identify services that dynamically generated responses, such as image-maps [Kevin Hughes] and server-side scripts [Rob McCool]. Work had also begun on intermediary components, in the form of proxies [ 79 ] and shared caches [ 59 ], but extensions to the protocols were needed in order for them to communicate reliably. The following sections describe the constraints added to the Web's architectural style in order to guide the extensions that form the modern Web architecture.

5.1.5 Uniform Interface

The central feature that distinguishes the REST architectural style from other network-based styles is its emphasis on a uniform interface between components ( Figure 5-6 ). By applying the software engineering principle of generality to the component interface, the overall system architecture is simplified and the visibility of interactions is improved. Implementations are decoupled from the services they provide, which encourages independent evolvability. The trade-off, though, is that a uniform interface degrades efficiency, since information is transferred in a standardized form rather than one which is specific to an application's needs. The REST interface is designed to be efficient for large-grain hypermedia data transfer, optimizing for the common case of the Web, but resulting in an interface that is not optimal for other forms of architectural interaction.

In order to obtain a uniform interface, multiple architectural constraints are needed to guide the behavior of components. REST is defined by four interface constraints: identification of resources; manipulation of resources through representations; self-descriptive messages; and, hypermedia as the engine of application state. These constraints will be discussed in Section 5.2 .

5.1.6 Layered System

In order to further improve behavior for Internet-scale requirements, we add layered system constraints ( Figure 5-7 ). As described in Section 3.4.2 , the layered system style allows an architecture to be composed of hierarchical layers by constraining component behavior such that each component cannot "see" beyond the immediate layer with which they are interacting. By restricting knowledge of the system to a single layer, we place a bound on the overall system complexity and promote substrate independence. Layers can be used to encapsulate legacy services and to protect new services from legacy clients, simplifying components by moving infrequently used functionality to a shared intermediary. Intermediaries can also be used to improve system scalability by enabling load balancing of services across multiple networks and processors.

The primary disadvantage of layered systems is that they add overhead and latency to the processing of data, reducing user-perceived performance [ 32 ]. For a network-based system that supports cache constraints, this can be offset by the benefits of shared caching at intermediaries. Placing shared caches at the boundaries of an organizational domain can result in significant performance benefits [ 136 ]. Such layers also allow security policies to be enforced on data crossing the organizational boundary, as is required by firewalls [ 79 ].

The combination of layered system and uniform interface constraints induces architectural properties similar to those of the uniform pipe-and-filter style ( Section 3.2.2 ). Although REST interaction is two-way, the large-grain data flows of hypermedia interaction can each be processed like a data-flow network, with filter components selectively applied to the data stream in order to transform the content as it passes [ 26 ]. Within REST, intermediary components can actively transform the content of messages because the messages are self-descriptive and their semantics are visible to intermediaries.

5.1.7 Code-On-Demand

The final addition to our constraint set for REST comes from the code-on-demand style of Section 3.5.3 ( Figure 5-8 ). REST allows client functionality to be extended by downloading and executing code in the form of applets or scripts. This simplifies clients by reducing the number of features required to be pre-implemented. Allowing features to be downloaded after deployment improves system extensibility. However, it also reduces visibility, and thus is only an optional constraint within REST.

The notion of an optional constraint may seem like an oxymoron. However, it does have a purpose in the architectural design of a system that encompasses multiple organizational boundaries. It means that the architecture only gains the benefit (and suffers the disadvantages) of the optional constraints when they are known to be in effect for some realm of the overall system. For example, if all of the client software within an organization is known to support Java applets [ 45 ], then services within that organization can be constructed such that they gain the benefit of enhanced functionality via downloadable Java classes. At the same time, however, the organization's firewall may prevent the transfer of Java applets from external sources, and thus to the rest of the Web it will appear as if those clients do not support code-on-demand. An optional constraint allows us to design an architecture that supports the desired behavior in the general case, but with the understanding that it may be disabled within some contexts.

5.1.8 Style Derivation Summary

REST consists of a set of architectural constraints chosen for the properties they induce on candidate architectures. Although each of these constraints can be considered in isolation, describing them in terms of their derivation from common architectural styles makes it easier to understand the rationale behind their selection. Figure 5-9 depicts the derivation of REST's constraints graphically in terms of the network-based architectural styles examined in Chapter 3.

5.2 REST Architectural Elements

The Representational State Transfer (REST) style is an abstraction of the architectural elements within a distributed hypermedia system. REST ignores the details of component implementation and protocol syntax in order to focus on the roles of components, the constraints upon their interaction with other components, and their interpretation of significant data elements. It encompasses the fundamental constraints upon components, connectors, and data that define the basis of the Web architecture, and thus the essence of its behavior as a network-based application.

5.2.1 Data Elements

Unlike the distributed object style [ 31 ], where all data is encapsulated within and hidden by the processing components, the nature and state of an architecture's data elements is a key aspect of REST. The rationale for this design can be seen in the nature of distributed hypermedia. When a link is selected, information needs to be moved from the location where it is stored to the location where it will be used by, in most cases, a human reader. This is unlike many other distributed processing paradigms [ 6 , 50 ], where it is possible, and usually more efficient, to move the "processing agent" (e.g., mobile code, stored procedure, search expression, etc.) to the data rather than move the data to the processor.

A distributed hypermedia architect has only three fundamental options: 1) render the data where it is located and send a fixed-format image to the recipient; 2) encapsulate the data with a rendering engine and send both to the recipient; or, 3) send the raw data to the recipient along with metadata that describes the data type, so that the recipient can choose their own rendering engine.

Each option has its advantages and disadvantages. Option 1, the traditional client-server style [ 31 ], allows all information about the true nature of the data to remain hidden within the sender, preventing assumptions from being made about the data structure and making client implementation easier. However, it also severely restricts the functionality of the recipient and places most of the processing load on the sender, leading to scalability problems. Option 2, the mobile object style [ 50 ], provides information hiding while enabling specialized processing of the data via its unique rendering engine, but limits the functionality of the recipient to what is anticipated within that engine and may vastly increase the amount of data transferred. Option 3 allows the sender to remain simple and scalable while minimizing the bytes transferred, but loses the advantages of information hiding and requires that both sender and recipient understand the same data types.

REST provides a hybrid of all three options by focusing on a shared understanding of data types with metadata, but limiting the scope of what is revealed to a standardized interface. REST components communicate by transferring a representation of a resource in a format matching one of an evolving set of standard data types, selected dynamically based on the capabilities or desires of the recipient and the nature of the resource. Whether the representation is in the same format as the raw source, or is derived from the source, remains hidden behind the interface. The benefits of the mobile object style are approximated by sending a representation that consists of instructions in the standard data format of an encapsulated rendering engine (e.g., Java [ 45 ]). REST therefore gains the separation of concerns of the client-server style without the server scalability problem, allows information hiding through a generic interface to enable encapsulation and evolution of services, and provides for a diverse set of functionality through downloadable feature-engines.

REST's data elements are summarized in Table 5-1 .

5.2.1.1 Resources and Resource Identifiers

The key abstraction of information in REST is a resource . Any information that can be named can be a resource: a document or image, a temporal service (e.g. "today's weather in Los Angeles"), a collection of other resources, a non-virtual object (e.g. a person), and so on. In other words, any concept that might be the target of an author's hypertext reference must fit within the definition of a resource. A resource is a conceptual mapping to a set of entities, not the entity that corresponds to the mapping at any particular point in time.

More precisely, a resource R is a temporally varying membership function M R (t) , which for time t maps to a set of entities, or values, which are equivalent. The values in the set may be resource representations and/or resource identifiers . A resource can map to the empty set, which allows references to be made to a concept before any realization of that concept exists -- a notion that was foreign to most hypertext systems prior to the Web [ 61 ]. Some resources are static in the sense that, when examined at any time after their creation, they always correspond to the same value set. Others have a high degree of variance in their value over time. The only thing that is required to be static for a resource is the semantics of the mapping, since the semantics is what distinguishes one resource from another.

For example, the "authors' preferred version" of an academic paper is a mapping whose value changes over time, whereas a mapping to "the paper published in the proceedings of conference X" is static. These are two distinct resources, even if they both map to the same value at some point in time. The distinction is necessary so that both resources can be identified and referenced independently. A similar example from software engineering is the separate identification of a version-controlled source code file when referring to the "latest revision", "revision number 1.2.7", or "revision included with the Orange release."

This abstract definition of a resource enables key features of the Web architecture. First, it provides generality by encompassing many sources of information without artificially distinguishing them by type or implementation. Second, it allows late binding of the reference to a representation, enabling content negotiation to take place based on characteristics of the request. Finally, it allows an author to reference the concept rather than some singular representation of that concept, thus removing the need to change all existing links whenever the representation changes (assuming the author used the right identifier).

REST uses a resource identifier to identify the particular resource involved in an interaction between components. REST connectors provide a generic interface for accessing and manipulating the value set of a resource, regardless of how the membership function is defined or the type of software that is handling the request. The naming authority that assigned the resource identifier, making it possible to reference the resource, is responsible for maintaining the semantic validity of the mapping over time (i.e., ensuring that the membership function does not change).

Traditional hypertext systems [ 61 ], which typically operate in a closed or local environment, use unique node or document identifiers that change every time the information changes, relying on link servers to maintain references separately from the content [ 135 ]. Since centralized link servers are an anathema to the immense scale and multi-organizational domain requirements of the Web, REST relies instead on the author choosing a resource identifier that best fits the nature of the concept being identified. Naturally, the quality of an identifier is often proportional to the amount of money spent to retain its validity, which leads to broken links as ephemeral (or poorly supported) information moves or disappears over time.

5.2.1.2 Representations

REST components perform actions on a resource by using a representation to capture the current or intended state of that resource and transferring that representation between components. A representation is a sequence of bytes, plus representation metadata to describe those bytes. Other commonly used but less precise names for a representation include: document, file, and HTTP message entity, instance, or variant.

A representation consists of data, metadata describing the data, and, on occasion, metadata to describe the metadata (usually for the purpose of verifying message integrity). Metadata is in the form of name-value pairs, where the name corresponds to a standard that defines the value's structure and semantics. Response messages may include both representation metadata and resource metadata: information about the resource that is not specific to the supplied representation.

Control data defines the purpose of a message between components, such as the action being requested or the meaning of a response. It is also used to parameterize requests and override the default behavior of some connecting elements. For example, cache behavior can be modified by control data included in the request or response message.

Depending on the message control data, a given representation may indicate the current state of the requested resource, the desired state for the requested resource, or the value of some other resource, such as a representation of the input data within a client's query form, or a representation of some error condition for a response. For example, remote authoring of a resource requires that the author send a representation to the server, thus establishing a value for that resource that can be retrieved by later requests. If the value set of a resource at a given time consists of multiple representations, content negotiation may be used to select the best representation for inclusion in a given message.

The data format of a representation is known as a media type [ 48 ]. A representation can be included in a message and processed by the recipient according to the control data of the message and the nature of the media type. Some media types are intended for automated processing, some are intended to be rendered for viewing by a user, and a few are capable of both. Composite media types can be used to enclose multiple representations in a single message.

The design of a media type can directly impact the user-perceived performance of a distributed hypermedia system. Any data that must be received before the recipient can begin rendering the representation adds to the latency of an interaction. A data format that places the most important rendering information up front, such that the initial information can be incrementally rendered while the rest of the information is being received, results in much better user-perceived performance than a data format that must be entirely received before rendering can begin.

For example, a Web browser that can incrementally render a large HTML document while it is being received provides significantly better user-perceived performance than one that waits until the entire document is completely received prior to rendering, even though the network performance is the same. Note that the rendering ability of a representation can also be impacted by the choice of content. If the dimensions of dynamically-sized tables and embedded objects must be determined before they can be rendered, their occurrence within the viewing area of a hypermedia page will increase its latency.

5.2.2 Connectors

REST uses various connector types, summarized in Table 5-2 , to encapsulate the activities of accessing resources and transferring resource representations. The connectors present an abstract interface for component communication, enhancing simplicity by providing a clean separation of concerns and hiding the underlying implementation of resources and communication mechanisms. The generality of the interface also enables substitutability: if the users' only access to the system is via an abstract interface, the implementation can be replaced without impacting the users. Since a connector manages network communication for a component, information can be shared across multiple interactions in order to improve efficiency and responsiveness.

All REST interactions are stateless. That is, each request contains all of the information necessary for a connector to understand the request, independent of any requests that may have preceded it. This restriction accomplishes four functions: 1) it removes any need for the connectors to retain application state between requests, thus reducing consumption of physical resources and improving scalability; 2) it allows interactions to be processed in parallel without requiring that the processing mechanism understand the interaction semantics; 3) it allows an intermediary to view and understand a request in isolation, which may be necessary when services are dynamically rearranged; and, 4) it forces all of the information that might factor into the reusability of a cached response to be present in each request.

The connector interface is similar to procedural invocation, but with important differences in the passing of parameters and results. The in-parameters consist of request control data, a resource identifier indicating the target of the request, and an optional representation. The out-parameters consist of response control data, optional resource metadata, and an optional representation. From an abstract viewpoint the invocation is synchronous, but both in and out-parameters can be passed as data streams. In other words, processing can be invoked before the value of the parameters is completely known, thus avoiding the latency of batch processing large data transfers.

The primary connector types are client and server. The essential difference between the two is that a client initiates communication by making a request, whereas a server listens for connections and responds to requests in order to supply access to its services. A component may include both client and server connectors.

A third connector type, the cache connector, can be located on the interface to a client or server connector in order to save cacheable responses to current interactions so that they can be reused for later requested interactions. A cache may be used by a client to avoid repetition of network communication, or by a server to avoid repeating the process of generating a response, with both cases serving to reduce interaction latency. A cache is typically implemented within the address space of the connector that uses it.

Some cache connectors are shared, meaning that its cached responses may be used in answer to a client other than the one for which the response was originally obtained. Shared caching can be effective at reducing the impact of "flash crowds" on the load of a popular server, particularly when the caching is arranged hierarchically to cover large groups of users, such as those within a company's intranet, the customers of an Internet service provider, or Universities sharing a national network backbone. However, shared caching can also lead to errors if the cached response does not match what would have been obtained by a new request. REST attempts to balance the desire for transparency in cache behavior with the desire for efficient use of the network, rather than assuming that absolute transparency is always required.

A cache is able to determine the cacheability of a response because the interface is generic rather than specific to each resource. By default, the response to a retrieval request is cacheable and the responses to other requests are non-cacheable. If some form of user authentication is part of the request, or if the response indicates that it should not be shared, then the response is only cacheable by a non-shared cache. A component can override these defaults by including control data that marks the interaction as cacheable, non-cacheable or cacheable for only a limited time.

A resolver translates partial or complete resource identifiers into the network address information needed to establish an inter-component connection. For example, most URI include a DNS hostname as the mechanism for identifying the naming authority for the resource. In order to initiate a request, a Web browser will extract the hostname from the URI and make use of a DNS resolver to obtain the Internet Protocol address for that authority. Another example is that some identification schemes (e.g., URN [ 124 ]) require an intermediary to translate a permanent identifier to a more transient address in order to access the identified resource. Use of one or more intermediate resolvers can improve the longevity of resource references through indirection, though doing so adds to the request latency.

The final form of connector type is a tunnel, which simply relays communication across a connection boundary, such as a firewall or lower-level network gateway. The only reason it is modeled as part of REST and not abstracted away as part of the network infrastructure is that some REST components may dynamically switch from active component behavior to that of a tunnel. The primary example is an HTTP proxy that switches to a tunnel in response to a CONNECT method request [ 71 ], thus allowing its client to directly communicate with a remote server using a different protocol, such as TLS, that doesn't allow proxies. The tunnel disappears when both ends terminate their communication.

5.2.3 Components

REST components, summarized in Table 5-3 , are typed by their roles in an overall application action.

A user agent uses a client connector to initiate a request and becomes the ultimate recipient of the response. The most common example is a Web browser, which provides access to information services and renders service responses according to the application needs.

An origin server uses a server connector to govern the namespace for a requested resource. It is the definitive source for representations of its resources and must be the ultimate recipient of any request that intends to modify the value of its resources. Each origin server provides a generic interface to its services as a resource hierarchy. The resource implementation details are hidden behind the interface.

Intermediary components act as both a client and a server in order to forward, with possible translation, requests and responses. A proxy component is an intermediary selected by a client to provide interface encapsulation of other services, data translation, performance enhancement, or security protection. A gateway (a.k.a., reverse proxy) component is an intermediary imposed by the network or origin server to provide an interface encapsulation of other services, for data translation, performance enhancement, or security enforcement. Note that the difference between a proxy and a gateway is that a client determines when it will use a proxy.

5.3 REST Architectural Views

Now that we have an understanding of the REST architectural elements in isolation, we can use architectural views [ 105 ] to describe how the elements work together to form an architecture. Three types of view--process, connector, and data--are useful for illuminating the design principles of REST.

5.3.1 Process View

A process view of an architecture is primarily effective at eliciting the interaction relationships among components by revealing the path of data as it flows through the system. Unfortunately, the interaction of a real system usually involves an extensive number of components, resulting in an overall view that is obscured by the details. Figure 5-10 provides a sample of the process view from a REST-based architecture at a particular instance during the processing of three parallel requests.

REST's client-server separation of concerns simplifies component implementation, reduces the complexity of connector semantics, improves the effectiveness of performance tuning, and increases the scalability of pure server components. Layered system constraints allow intermediaries--proxies, gateways, and firewalls--to be introduced at various points in the communication without changing the interfaces between components, thus allowing them to assist in communication translation or improve performance via large-scale, shared caching. REST enables intermediate processing by constraining messages to be self-descriptive: interaction is stateless between requests, standard methods and media types are used to indicate semantics and exchange information, and responses explicitly indicate cacheability.

Since the components are connected dynamically, their arrangement and function for a particular application action has characteristics similar to a pipe-and-filter style. Although REST components communicate via bidirectional streams, the processing of each direction is independent and therefore susceptible to stream transducers (filters). The generic connector interface allows components to be placed on the stream based on the properties of each request or response.

Services may be implemented using a complex hierarchy of intermediaries and multiple distributed origin servers. The stateless nature of REST allows each interaction to be independent of the others, removing the need for an awareness of the overall component topology, an impossible task for an Internet-scale architecture, and allowing components to act as either destinations or intermediaries, determined dynamically by the target of each request. Connectors need only be aware of each other's existence during the scope of their communication, though they may cache the existence and capabilities of other components for performance reasons.

5.3.2 Connector View

A connector view of an architecture concentrates on the mechanics of the communication between components. For a REST-based architecture, we are particularly interested in the constraints that define the generic resource interface.

Client connectors examine the resource identifier in order to select an appropriate communication mechanism for each request. For example, a client may be configured to connect to a specific proxy component, perhaps one acting as an annotation filter, when the identifier indicates that it is a local resource. Likewise, a client can be configured to reject requests for some subset of identifiers.

REST does not restrict communication to a particular protocol, but it does constrain the interface between components, and hence the scope of interaction and implementation assumptions that might otherwise be made between components. For example, the Web's primary transfer protocol is HTTP, but the architecture also includes seamless access to resources that originate on pre-existing network servers, including FTP [ 107 ], Gopher [ 7 ], and WAIS [ 36 ]. Interaction with those services is restricted to the semantics of a REST connector. This constraint sacrifices some of the advantages of other architectures, such as the stateful interaction of a relevance feedback protocol like WAIS, in order to retain the advantages of a single, generic interface for connector semantics. In return, the generic interface makes it possible to access a multitude of services through a single proxy. If an application needs the additional capabilities of another architecture, it can implement and invoke those capabilities as a separate system running in parallel, similar to how the Web architecture interfaces with "telnet" and "mailto" resources.

5.3.3 Data View

A data view of an architecture reveals the application state as information flows through the components. Since REST is specifically targeted at distributed information systems, it views an application as a cohesive structure of information and control alternatives through which a user can perform a desired task. For example, looking-up a word in an on-line dictionary is one application, as is touring through a virtual museum, or reviewing a set of class notes to study for an exam. Each application defines goals for the underlying system, against which the system's performance can be measured.

Component interactions occur in the form of dynamically sized messages. Small or medium-grain messages are used for control semantics, but the bulk of application work is accomplished via large-grain messages containing a complete resource representation. The most frequent form of request semantics is that of retrieving a representation of a resource (e.g., the "GET" method in HTTP), which can often be cached for later reuse.

REST concentrates all of the control state into the representations received in response to interactions. The goal is to improve server scalability by eliminating any need for the server to maintain an awareness of the client state beyond the current request. An application's state is therefore defined by its pending requests, the topology of connected components (some of which may be filtering buffered data), the active requests on those connectors, the data flow of representations in response to those requests, and the processing of those representations as they are received by the user agent.

An application reaches a steady-state whenever it has no outstanding requests; i.e., it has no pending requests and all of the responses to its current set of requests have been completely received or received to the point where they can be treated as a representation data stream. For a browser application, this state corresponds to a "web page," including the primary representation and ancillary representations, such as in-line images, embedded applets, and style sheets. The significance of application steady-states is seen in their impact on both user-perceived performance and the burstiness of network request traffic.

The user-perceived performance of a browser application is determined by the latency between steady-states: the period of time between the selection of a hypermedia link on one web page and the point when usable information has been rendered for the next web page. The optimization of browser performance is therefore centered around reducing this communication latency.

Since REST-based architectures communicate primarily through the transfer of representations of resources, latency can be impacted by both the design of the communication protocols and the design of the representation data formats. The ability to incrementally render the response data as it is received is determined by the design of the media type and the availability of layout information (visual dimensions of in-line objects) within each representation.

An interesting observation is that the most efficient network request is one that doesn't use the network. In other words, the ability to reuse a cached response results in a considerable improvement in application performance. Although use of a cache adds some latency to each individual request due to lookup overhead, the average request latency is significantly reduced when even a small percentage of requests result in usable cache hits.

The next control state of an application resides in the representation of the first requested resource, so obtaining that first representation is a priority. REST interaction is therefore improved by protocols that "respond first and think later." In other words, a protocol that requires multiple interactions per user action, in order to do things like negotiate feature capabilities prior to sending a content response, will be perceptively slower than a protocol that sends whatever is most likely to be optimal first and then provides a list of alternatives for the client to retrieve if the first response is unsatisfactory.

The application state is controlled and stored by the user agent and can be composed of representations from multiple servers. In addition to freeing the server from the scalability problems of storing state, this allows the user to directly manipulate the state (e.g., a Web browser's history), anticipate changes to that state (e.g., link maps and prefetching of representations), and jump from one application to another (e.g., bookmarks and URI-entry dialogs).

The model application is therefore an engine that moves from one state to the next by examining and choosing from among the alternative state transitions in the current set of representations. Not surprisingly, this exactly matches the user interface of a hypermedia browser. However, the style does not assume that all applications are browsers. In fact, the application details are hidden from the server by the generic connector interface, and thus a user agent could equally be an automated robot performing information retrieval for an indexing service, a personal agent looking for data that matches certain criteria, or a maintenance spider busy patrolling the information for broken references or modified content [ 39 ].

5.4 Related Work

Bass, et al. [ 9 ] devote a chapter on architecture for the World Wide Web, but their description only encompasses the implementation architecture within the CERN/W3C developed libwww (client and server libraries) and Jigsaw software. Although those implementations reflect many of the design constraints of REST, having been developed by people familiar with the Web's architectural design and rationale, the real WWW architecture is independent of any single implementation. The modern Web is defined by its standard interfaces and protocols, not how those interfaces and protocols are implemented in a given piece of software.

The REST style draws from many preexisting distributed process paradigms [ 6 , 50 ], communication protocols, and software fields. REST component interactions are structured in a layered client-server style, but the added constraints of the generic resource interface create the opportunity for substitutability and inspection by intermediaries. Requests and responses have the appearance of a remote invocation style, but REST messages are targeted at a conceptual resource rather than an implementation identifier.

Several attempts have been made to model the Web architecture as a form of distributed file system (e.g., WebNFS) or as a distributed object system [ 83 ]. However, they exclude various Web resource types or implementation strategies as being "not interesting," when in fact their presence invalidates the assumptions that underlie such models. REST works well because it does not limit the implementation of resources to certain predefined models, allowing each application to choose an implementation that best matches its own needs and enabling the replacement of implementations without impacting the user.

The interaction method of sending representations of resources to consuming components has some parallels with event-based integration (EBI) styles. The key difference is that EBI styles are push-based. The component containing the state (equivalent to an origin server in REST) issues an event whenever the state changes, whether or not any component is actually interested in or listening for such an event. In the REST style, consuming components usually pull representations. Although this is less efficient when viewed as a single client wishing to monitor a single resource, the scale of the Web makes an unregulated push model infeasible.

The principled use of the REST style in the Web, with its clear notion of components, connectors, and representations, relates closely to the C2 architectural style [ 128 ]. The C2 style supports the development of distributed, dynamic applications by focusing on structured use of connectors to obtain substrate independence. C2 applications rely on asynchronous notification of state changes and request messages. As with other event-based schemes, C2 is nominally push-based, though a C2 architecture could operate in REST's pull style by only emitting a notification upon receipt of a request. However, the C2 style lacks the intermediary-friendly constraints of REST, such as the generic resource interface, guaranteed stateless interactions, and intrinsic support for caching.

5.5 Summary

This chapter introduced the Representational State Transfer (REST) architectural style for distributed hypermedia systems. REST provides a set of architectural constraints that, when applied as a whole, emphasizes scalability of component interactions, generality of interfaces, independent deployment of components, and intermediary components to reduce interaction latency, enforce security, and encapsulate legacy systems. I described the software engineering principles guiding REST and the interaction constraints chosen to retain those principles, while contrasting them to the constraints of other architectural styles.

The next chapter presents an evaluation of the REST architecture through the experience and lessons learned from applying REST to the design, specification, and deployment of the modern Web architecture. This work included authoring the current Internet standards-track specifications of the Hypertext Transfer Protocol (HTTP/1.1) and Uniform Resource Identifiers (URI), and implementing the architecture through the libwww-perl client protocol library and Apache HTTP server.

An Analysis of Public REST Web Service APIs

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2023 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

International Conference on Web Engineering

ICWE 2016: Web Engineering pp 21–39 Cite as

REST APIs: A Large-Scale Analysis of Compliance with Principles and Best Practices

• Carlos Rodríguez 16 ,
• Marcos Baez 16 ,
• Florian Daniel 17 ,
• Fabio Casati 16 ,
• Juan Carlos Trabucco 18 ,
• Luigi Canali 18 &
• Gianraffaele Percannella 18  
• Conference paper
• First Online: 25 May 2016

5387 Accesses

41 Citations

2 Altmetric

Part of the Lecture Notes in Computer Science book series (LNISA,volume 9671)

Quickly and dominantly, REST APIs have spread over the Web and percolated into modern software development practice, especially in the Mobile Internet where they conveniently enable offloading data and computations onto cloud services. We analyze more than 78 GB of HTTP traffic collected by Italy’s biggest Mobile Internet provider over one full day and study how big the trend is in practice, how it changed the traffic that is generated by applications, and how REST APIs are implemented in practice. The analysis provides insight into the compliance of state-of-the-art APIs with theoretical Web engineering principles and guidelines, knowledge that affects how applications should be developed to be scalable and robust. The perspective is that of the Mobile Internet.

• REST principles
• Mobile internet

Download conference paper PDF

1 Introduction

By now, Web applications leveraging on remote APIs or services, service-oriented applications or service compositions [ 21 ], mashups [ 5 ], mobile applications built on top of cloud services and similar web technologies are state of the art. They all have in common the heavy use of functionality, application logic and/or data sourced from the own backend or third parties via Web services or APIs that provide added value and are accessible worldwide with only little development effort. The continuous and sustained growth of ProgrammableWeb’s API directory ( http://www.programmableweb.com/apis/directory ) is only the most immediate evidence of the success that Web services and APIs have had and are having among developers. On the one hand, today it is hard to imagine a Web application or a mobile app that does not leverage on some kind of remote resource, be it a Google Map or some application-specific, proprietary functionality. On the other hand, to some companies today service/API calls represent the equivalent of page visits in terms of business value.

Two core types of remote programming resources have emerged over the years: SOAP/WSDL Web services [ 21 ] and REST APIs [ 6 ]. While the former can rely on a very rich set of standards and reference specifications, and developers know well how to use WSDL [ 4 ] to describe a service and SOAP [ 3 ] to exchange messages with clients, REST APIs do not have experienced this kind of standardization (we specifically refer to JSON/XML APIs for software agents and exclude web apps for human actors). Indeed, REST is an architectural style and a guideline of how to use HTTP [ 7 ] for the development of highly scalable and robust APIs. While the freedom left by this choice is one of the reasons for the fast uptake of REST, it is also a reasons why everybody interprets REST in an own way and follows guidelines and best practices only partially, if at all.

It goes without saying that even small differences in the interpretation of the principles and guidelines underlying REST APIs can turn into a tedious and intricate puzzle to the developer that has to integrate multiple APIs that each work differently, although expected to behave similarly. For instance, while one provider may accompany an own API with a suitable WADL [ 10 ] description, another provider may instead not provide any description at all and require interested clients to navigate through and explore autonomously the resources managed by the API. Of course, if instead all APIs consistently followed the same principles and guidelines, this would result in design features (e.g., decoupling, reusability, tolerance to evolution) that would directly translate into savings in development and maintainance costs and time [ 18 , 23 ].

With this paper, we provide up-to-date insight into how well or bad the principles and guidelines of the REST architectural style are followed by looking at the problem from the mobile perspective. We thus take an original point of view: we analyze more than 78 GB of plain HTTP traffic collected by Italy’s biggest Mobile Internet (MI) provider, Telecom Italia, identify which of the individual HTTP calls are targeted at REST APIs, and characterize the usage patterns that emerge from the logged data so as to compare them with guidelines and principles. We further use the maturity model by Richardson [ 8 ], which offers an interesting way to look at REST in increasing levels of architectural gains, to distinguish different levels of compliance with the principles. The dataset we can rely on allows us, at the same time, to look at how conventional Web applications leverage on REST APIs as well as to bring in some insights regarding the use of APIs in the Mobile Internet. Concretely, the contributions of this paper are as follows:

We descriptively characterize a dataset of more than 78 GB of HTTP requests corresponding to one full day of Mobile Internet traffic generated by almost 1 million subscribers.

From the core principles and guidelines of REST and the structure of the dataset, we derive a set of heuristics and metrics that allow us to quantitatively describe the API ecosystem that emerges from the data.

We analyze the results, study how well the data backs the principles and guidelines of REST, and discuss how the respective findings may impact API maintainability and development.

The paper is structured in line with these contributions. We first recap the theoretical principles and guidelines that we want to study in this paper (Sect.  2 ). Next, we introduce the dataset we analyzed and how we collected it (Sect.  3 ) and discuss its key features (Sect.  4 ). Then, we specifically focus on the REST APIs (Sect.  5 ) and conclude the paper with an overview of related works and our final considerations on the findings (Sects.  6 and 7 ).

2 REST APIs

The Representational State Transfer (REST) architectural style [ 6 ] defines a set of rules for the design of distributed hypermedia systems that have guided the design and development of the Web as we know it. Web services following the REST architectural style are referred to as RESTful Web services , and the programmatic interfaces of these services as REST APIs . The principles governing the design of REST APIs are in big part the result of architectural choices of the Web aimed at fostering scalability and robustness of networked, resource-oriented systems based on HTTP [ 7 ]. The core principles are [ 6 , 23 ]:

Resource addressability . APIs manage and expose resources representing domain concepts; each resource is uniquely identified and addressable by a suitable Uniform Resource Identifier (URI).

Resource representations . Clients do not directly know the internal format and state of resources; they work with resource representations (e.g., JSON or XML) that represent the current or intended state of a resource. The declaration of content-types in the headers of HTTP messages enables clients and servers to properly process representations.

Uniform interface . Resources are accessed and manipulated using the standard methods defined by the HTTP protocol ( Get , Post , Put , etc.). Each method has its own expected, standard behavior and standard status codes.

Statelessness . Interactions between a client and an API are stateless, meaning that each request contains all the necessary information to be processed by the API; no interaction state is kept on the server.

Hypermedia as the engine of state . Resources as domain concepts can be related to other resources. Links between resources (included in their representations) allow clients to discover and navigate relationships and to maintain interaction state.

Together, these principles explain the name “representational state transfer”: interaction state is not stored on the server side; it is carried (transferred) by each request from the client to the server and encoded inside the representation of the resource the request refers to.

2.1 Best Practices for Development

Along with the general principles introduced above, a set of implementation best practices have emerged to guide the design of quality APIs [ 16 , 19 , 22 , 23 ]. These best practices address the main design aspects in REST APIs: (i) the modeling of resources, (ii) the identification of resources and the design of resource identifiers (URIs), (iii) the representation of resources, (iv) the definition of (HTTP) operations on resources, and (v) the interlinking of resources. We overview these best practices in the following; a summary with examples is shown in Table  1 .

Resource modeling. REST APIs can manage different types of resources: documents for single instances of resources, collections for groups of resources, and controllers for actions that cannot logically be mapped to the standard HTTP methods [ 16 ]. While modeling resources for REST APIs is not fundamentally different from modeling classes in OO programming or entities in data modeling, there are a couple of recommended naming practices that are typical of REST APIs: singular nouns for documents, plural nouns for collections, and verbs only for controllers [ 16 ], no CRUD names in URLs [ 16 , 22 ], no transparency of server-side implementation technologies (e.g., PHP, JSP) ( http://www.ibm.com/developerworks/library/ws-restful/ ).

Resource identification. Resource identifiers should conform with the URI format, consisting of a scheme, authority, path, query, and fragment [ 2 ]. In the case of Web-accessible REST APIs, the URIs are typically URLs (Uniform Resource Locators) that tell clients how to locate the APIs. In order to improve the readability of URLs, it is recommended to use hyphens instead of underscores, lowercase letters in paths, “api” as part of the domain, and avoid the trailing forward slash [ 16 ]. In addition, in its purest form, REST services should avoid declaring API versions in the URL [ 16 ].

Resource representation. Resources can support alternative representations (e.g., XML, JSON) and serve different clients with different formats. Which representation to serve should be negotiated at runtime, with the client expressing its desired representation using the HTTP Accept header instruction. This fosters reusability, interoperability and loose-coupling [ 22 ]. APIs should therefore use content negotiation instead of file extensions to specify formats (e.g., .json or .xml ). In addition, it is recommended that APIs support (valid) JSON among their representation alternatives [ 16 , 22 ].

Operations. To manage resources, REST APIs should rely on the uniform set of operations ( Post , Get , Put , Delete , Options , Head ) defined by the HTTP standard [ 7 ] and comply with their standardized semantics:

Post should be used to create new resources within a collection.

Get should be used to retrieve a representation of a resource.

Put should be used to update or create resources.

Delete should be used to remove a resource from its parent.

Options should be used to retrieve the available interactions of a resource.

Head should be used to retrieve metadata of the current state of a resource.

REST APIs should thus never tunnel requests through Get or Post , e.g., by specifying the actual operation as a parameter or as part of the resource name.

2.2 Assessing REST Compliance

Next to the lower-level development best practices, concrete APIs may follow the very principles underlying REST to different extents. The maturity model by Richardson [ 8 ] offers a way to explain the respective degree of compliance by means of different levels of maturity:

Level 0 : At this level, APIs work by tunneling requests through a single endpoint (URL) using one HTTP method. Examples of services working at this level are XML-RPC and those SOAP/WSDL services that transmit all communications as HTTP Post requests and use HTTP purely as transport protocol. Yet, also some REST APIs adopt this technique.

Level 1 : At this level, instead of using a single endpoint, functionality exposed by the API is split over multiple resources , which increases the addressability of the API and facilitates consumption. However, services at Level 1 still make use of payload data or the URL to identify operations.

Level 2 : APIs at this level make proper use of the HTTP methods and status codes for each resource and correctly follow the uniform interface principle.

Level 3 : APIs at this level embrace the notion of hypermedia . Thus, not only resources can be accessed through a uniform interface but their relationships can be discovered and explored via suitable links.

Each level of compliance comes with greater benefits in terms of quality and ease of use by the developer familiar with REST. We will come back to these levels when analyzing the adherence of APIs to the principles and best practices.

3 Mobile Telco Infrastructure and Dataset

In order to study how well the state-of-the-art landscape of REST APIs complies with the introduced principles and guidelines, in this paper we rely on a dataset of 78 GB of plain HTTP traffic collected by Italys biggest Mobile Internet (MI) provider, Telecom Italia. To understand the nature and provenance of the dataset, Fig.  1 provides a functional overview of the underlying cellular network architecture (upper part) and of how data was collected (lower part).

The cellular network uses 2G (GSM/GPRS), 3G (UMTS) and 4G (LTE) base stations (Node B) for the connection of mobile devices. The Radio Network Controllers (RNCs) control the base stations and connect to the Serving GPRS Support Nodes (SGSNs) that provide packet-switched access to the core network of the operator within their service areas. Via the core network, the SGSNs are connected with the Gateway GPRS Support Nodes (GGSNs) that mediate between the core network of the operator and external packet-switched networks, in our case the Internet. The GGSNs also assign the IP addresses to the devices connected to the Internet through the operator’s own network.

Cellular network architecture with probes for the collection of Mobile Internet usage data and an excerpt of the structure of the data studied in this article.

If a mobile device issues an HTTP request to a server accessible over the Internet, the request traverses all the described components from left to right. Special hardware probes tap into the connection between the SGSN and the GGSN to intercept raw traffic. The probes forward the traffic to multiple, parallel data collectors that filter the intercepted data by purpose (we specifically focus on network usage and HTTP traffic) and produce purpose-specific log files as output; each file contains approximately 15 min of traffic. For our analysis, a pre-processing of the files is needed to join the HTTP traffic records with the network usage records, so as to be able to correlate traffic with network usage properties like cell IDs or data sizes.

The result is a set of joint, enriched HTTP traffic files of which Fig. 1 shows an excerpt of the data structure: Sub_Id and IP are the subscriber identifier and IP address (both fully anonymized), StartTime and EndTime delimit the HTTP transaction as registered by the cellular network, URL contains the complete URL requested by the mobile device, HTTP_Head contains the full header of the HTTP request, Bytes contains the size of the data uploaded/downloaded, and Cell_Id uniquely identifies the base station the device was connected to.

The available dataset was collected throughout the full day of 14 October (Wednesday) by one data collector located in the metropolitan area of Milan, Italy. The average amount of HTTP traffic recorded per day is about 150 GB (about 340 mln individual HTTP requests), the usage data is in the order of 200 GB/day; the enriched HTTP traffic files amount to approximately 180 GB/day. The pre-processor joining the HTTP traffic and network usage files is implemented by the TILab software group in Trento using RabbitMQ ( https://www.rabbitmq.com ) for the parallel processing of chunks of input data and Redis ( http://redis.io ) for in-memory data caching of joined tuples to be added to the enriched HTTP traffic files in output.

Please note that, in line with similar Internet usage studies [ 1 ], personal identifiers were anonymized prior to the study, and only aggregated values are reported. Data are stored on in-house servers and password protected. Before publication, the work was checked by Telecom for compliance with Italian Law D.Lgs 196/2003 (which implements the EU Directive on Privacy and Electronic Communications of 2002), Telecom’s own policies, and the NDA signed between Telecom and University of Trento.

4 Mobile Internet Traffic Analysis

We start our analysis of the use of REST APIs with a set of descriptive statistics about the available dataset as a whole. We recall that the data contain all HTTP requests recorded by the data collector over one full day of usage, including regular Web browsing activities. The analysis of the dataset provides an up-to-date picture of the Mobile Internet and informs the design of heuristics for the identification of those calls that instead involve APIs only (next section).

It is important to note that our analysis is based on HTTP traffic only and, for instance, does not take into account HTTPS traffic, streaming of audio/video media, or other protocols. As for the quality of the data analyzed, the data pre-processor’s data joining logic has proven to have an approximate success rate of 90 % (due to diverse imprecisions in the input data); we could however not identify any systematic bias in the dataset due to failed joins.

4.1 HTTP Requests and Responses

Figure  2 summarizes the key characteristics of the dataset we leverage on in this paper. Figure  2 (a) reports on the different HTTP methods (also called “verbs”) used by the recorded HTTP requests, along with the respective count. We can see that the two most commonly used methods ( Get and Post ) dominate the traffic in today’s Mobile Internet, followed by other methods such as Connect , Head , Put , Options and Delete . The less common methods Propfind and Proppatch are used by Web Distributed Authoring and Versioning (WebDAV), an extension of HTTP for web content authoring operations (see RFC 2518 [ 9 ]). Source (used by the Icecast multimedia streaming protocol), Dvrget and Dvrpost (used for multimedia/multipart content and streaming over HTTP), and List are other non-standard HTTP methods.

The identified usage of HTTP methods provide a first indication of the potential compliance of the RESTful APIs with the REST architectural style guidelines [ 6 ], which, as we have seen earlier, advocate the use not only of Get and Post , but also of Put , Delete , Options , Head , etc. for the implementation of what is called the “uniform interface” of REST APIs. Our dataset shows that by now these request methods are not only being used by some APIs, but have turned into state of the art.

In this respect, it is good to keep in mind that the mobile app market is largely characterized by applications that heavily leverage on Web APIs to provide their users with mobile access to large content repositories and highly scalable computing power, two resources that are typically limited on mobile devices. Since our dataset captures Mobile Internet usage, there may be a bias toward a more rich use of HTTP methods. On the other hand, it is important to note that the Connect methods are used to establish HTTPS connections, that is to switch from plain HTTP to its encrypted counterpart HTTPS. Once a communication switches from HTTP to HTTPS (e.g., when a user logs in to Facebook) we are no longer able to intercept tunneled HTTP requests and, hence, to follow the conversation. The estimation of the telco operator is that, of all the mobile internet traffic, around 25–30% corresponds to HTTPS traffic. We acknowledge the lack of such type of traffic as a limitation of our dataset.

Descriptive statistics of the available dataset characterizing state-of-the-art Mobile Internet traffic as of October 2015.

Figure  2 (b) illustrates the counts of the HTTP response codes corresponding to the requests in Fig.  2 (a). According to the figure, the recorded requests feature a rich and varied usage of HTTP response codes. Responses are dominated by successful and redirection operations (2xx and 3xx codes), and errors (4xx and 5xx) are mainly due to clients requesting resources not found on the server (404) or forbidden to the client (403). In 2005, Bhole and Popescu [ 24 ] did a similar analysis of HTTP response codes and identified only 5 different codes in their dataset, with status code 200 representing 88 % of the analyzed traffic – despite the HTTP protocol specification (version 1.1) dating back to 1999 [ 7 ]. In other words, after approximately one decade HTTP responses are characterized today by a much richer use of response codes and APIs that effectively work with the standard semantics of both request methods and response codes.

Figure  2 (c) looks more detailedly into the different HTTP request methods and shows how much data is transmitted/received per method. Overall, the median of transmitted data is 1463 bytes, while the median of received data is 1643 bytes. The same numbers approximately hold for all methods, except for the Source method, which presents significantly higher values; we recall that the method is used by Icecast to stream multimedia content.

In 1995, Mah [ 13 ] showed that the median HTTP response length was about 2 KB. Pang et al. [ 20 ] registered a similar response length in 2005, and Maier et al. [ 15 ] approximately confirm analogous numbers in 2010. In the end of 2015, our dataset too confirms a similar median response length. This almost stable picture is somehow surprising, as over the last years we all have witnessed a Web that has grown more complex, in terms of both content and functionality. On the other hand, Mah also showed that in 1995 the median HTTP request length was about 240 bytes [ 13 ], while our dataset presents a median request length of about 1.5 KB. This change of the length of the requests must be explained by a different use of the Internet in upload between the two dates. In fact, from 1995 to today, the Web has evolved from Web 1.0 to Web 2.0, that is, from mono-directional content consumption to fully bidirectional content co-creation. The increase of request lengths provides evidences of this paradigm shift. A confirmation of this, however, would require an own, purposely designed study.

4.2 Media Type Usage

“Media types” are the generic Web synonym of “representations” in REST. Studying the media types returned by the HTTP requests allows one therefore to obtain a first indication of which representations state-of-the-art APIs use. Figure  2 (d) shows the ten most used media types in our dataset. Keeping in mind that the dataset contains generic Web traffic (not only API traffic), it is of no surprise to find text/html on the first place, followed by image/jpeg and image/gif . More surprising is that the data format application/json is already on the forth position, while text/xml is only on the ninth position. As can further be seen in the figure, both text/javascript and application/javascript refer to the same media type and, hence, the naming of the media types is not consistent throughout the different applications and/or APIs. In fact, it is good to note that the figure only shows the ten most used media type declarations; overall, the dataset contains 1134 different media type declarations.

The two media types that are of particular interest in this paper are of course JSON and XML, as these are meant for machine consumption and therefore refer to the invocation of an API or service. We exploit this property later on to identify calls to APIs among the huge amount of calls in the dataset. SOAP web services [ 21 ] too transfer XML-encoded data, yet the respective XML-encoded SOAP envelope is always associated with the media type application/soap+xml and transmitted via HTTP Post requests. It is therefore easy to distinguish calls to SOAP web services from potential calls to REST APIs.

4.3 User Agents

Finally, with Fig.  2 (e) we would like to shed some light on the user agents used to issue the requests logged in our dataset. The figure again shows the ten most used user agents from a total of 57571 different user agent declarations. On the first position, we find Mozilla/5.0 with an extraordinary predominance. To understand this result, it is important to notice that the user agent string in the header of HTTP requests can be assigned arbitrarily by the user agents themselves. And this is what happens in practice, as nicely explained by Aaron Andersen in his blog http://webaim.org/blog/user-agent-string-history/ : in order to prevent user agent sniffing and being discriminated, most modern Web browsers declare to be compatible with Mozilla/5.0 (even Internet Explorer, Edge, Safari and similar). More interesting to our own analysis are the user agents Dalvik (Android virtual machine), Android and Windows that testify the presence of mobile devices, while the user agents Instagram and MicroMessenger represent native mobile apps able to issue HTTP requests. Indeed, a closer inspection of our dataset revealed that 40.8 % of the traffic corresponds to native apps, while the rest 59.2 % is traffic generated from mobile, web browsers. As a follow up, future work we would like explore these two worlds with an own, dedicated study to understand whether and how they differ from each other.

5 REST API Analysis

Given our dataset, which can be seen as a generic dump of HTTP requests that interleaves requests directed toward APIs for machine consumption with requests directed toward Web applications for human consumption, the first problem to solve is identifying which requests actually refer to the former. This is necessary to be able to effectively focus the analysis on APIs for software agents (from now on simply APIs) and not to be distracted by regular Web navigation activities. Given the limited amount of information available about the recorded HTTP requests, the problem is not trivial and requires the application of API-specific heuristics.

Recalling Fig.  2 (d), we remember that among the top-10 media types used in our dataset we have JSON and XML, which are typical data formats for the exchange of data between software agents. It is thus reasonable to assume that requests returning any of these two media types are directed toward APIs. In order to identify such requests, we considered only those requests that contain the strings ‘json’ and ‘xml’ in their media type declaration. Examples of these include the common media types application/xml and application/json , but also less common media types such as application/vnd.nokia.ent.events+json and application/vnd.wap.xhtml+xml . The total number of such requests in our dataset is 18.2 million, 9.3 million for JSON and 8.9 million for XML.

Size in bytes for JSON and XML payloads, and media type distribution by host

In order to assure that these requests really return JSON and XML and to characterize the typical responses, we sampled all JSON and XML requests independently and representatively for the whole dataset using a 95 % confidence level and a confidence interval of 3. This corresponds to 1067 requests to the corresponding, presumed APIs randomly picked for both media types to obtain their payloads. Figure  3 (a) shows the cumulative density function of the payload sizes. The medians are 1545 and 2606 bytes, respectively, for JSON and XML.

We also checked the formal validity of the payloads. Checks were performed using Python’s internal libraries, which reported that 75 % and 76 % of the requests contained valid JSON and XML, respectively. The main reasons for invalid payloads were either empty payloads or, in the case of declared JSON payloads, the presence of JSONP callbacks (JSON wrapped in Javascript code) instead. As for the empty payloads, an inspection of the respective HTTP status codes reveals that most of them are explained by 4xx and 5xx error codes, that is, by resources that no longer exist or are not addressable on the server or because of session expiration. Overall, the counts of the status codes (in parenthesis) in the sample are: for JSON 1xx (0), 2xx (1204), 3xx (1), 4xx (243) and 5xx (53), and for XML 1xx (0), 2xx (1280), 3xx (0), 4xx (233) and 5xx (2).

The next step toward the identification of APIs would be deciding which concrete URLs serve as APIs end/entry points (e.g., api.server.org/universities ), starting from where clients can start exploring the APIs. Doing so is however not feasible without inspecting each API individually. We thus limit our analysis in this section to individual HTTP requests, without trying to infer API endpoints.

Given an HTTP request, the options for end points may range from the plain host name (e.g., api.server.org ) to the full URL at hand (e.g., api.server.org/universities/45/people/3 ). We discard this last option as too fine-grained, while, ideally, APIs should be accessible through a dedicated host name not used for other purposes. This would make the host name an identifier.

We tested this assumption: Using the same sample of 1067 requests as above, we identified the respective individual host names (incidentally precisely 1000) and went back to the full dataset recorded by the data collector to retrieve all media types that are accessible through these host names. If the host names were used only to provide API access, the media types would all be media types oriented toward software agents. In order to keep the computation manageable, we used a 15  min time slot of the full dataset collected during a high traffic hour. The slot contains a total of 3.2 million requests that, when joined with the 1000 different host names, corresponds to 3.2 billion comparisons. Figure  3 (b) shows the relative frequency for the top-10 media types identified. The media type aplication/json has the highest frequency, followed by text/html , text/xml and others. The presence of text/html , text/css and text/javascript indicates that through the same host names also content oriented toward human agents (Web sites) is delivered, not only content oriented toward software agents. Hence, we conclude that host names are not good API identifiers in general.

5.1 Compliance with Design Best Practices

Next, we specifically focus on the set of 18.2 million API requests identified previously and study how well the designers of the respective APIs followed the design principles and best practices introduced in Sect.  2 . We define a set of heuristics based on the request metadata available in the dataset as well as on the payloads we obtained from our representative sample of API invocations, in order to derive empirical evidence of compliance (or not).

Since some of the best practices as well as the maturity levels discussed in the next section do not apply to individual HTTP requests (which would be too fine-grained), we group requests by host names. This means that rather than studying the compliance of APIs or HTTP requests we study that of API providers, in that we look at the full traffic toward the APIs accessible under one and a same host name. Differently from the data underlying Fig.  3 (b), here we only focus on requests targeted toward JSON and XML resources and, hence, we are sure we study API-related traffic only.

The heuristics are summarized in Table  2 , implemented in JavaScript for node.js and Python, and the respective code is available on https://github.com/mbaezpy/api-analysis . For instance, rUnderscore uses a regular expression to tell whether an invoked URL contains underscores; we recall that the guideline is instead to use hyphens. rLowercase checks whether URLs comply with the guideline to use only lowercase letter, while rSlash checks that URLs don’t end with a final slash character. We refer the interested reader to the online resource for the concrete implementation of all heuristics.

Compliance of APIs with best practices and maturity levels of API providers.

Figure  4 (a) illustrates the mean, median and standard deviation of the compliance of the identified host names with each of the heuristics. For example, if we take heuristic rUnderscore , we can see that, on average, 75 % of the resources accessible through a host comply with this heuristic, with a median of 100 % and a standard deviation of 41 %, approximately. The figure shows that all except one heuristic ( rApiInDomain ) have a median of 100 %, and that they reached means higher than 95 %, the exceptions being rUnderscore , rLowercase , rSlash and rApiInDomain as well as rHideExtension and rFormatExtension .

Overall, these results are better than we expected. The lower compliance with the former four heuristics is not major issue that affects the quality of the actual service provided through an API; they refer to naming conventions, which may or may not be shared by all developers. However, the still rather high compliance with the first three heuristics tells that most of the developers actually do follow the best practice, while they don’t seem to like the use of “api” in the URL (consistently with the finding above that host names typically intermix content for human and software agents). The low compliance with the heuristics rHideExtension and rFormatExtension , instead, may have a negative effect on the maintainability and future evolvability of APIs. In fact, making implementation technologies explicit in the URL (e.g., the file suffix .php ) hinders the switch from one server-side implementation technology to another (e.g., node.js). By the same token, showing resource extensions (e.g., .json ) prevents content negotiation between client and server to agree on which representation format to exchange (e.g., XML instead of JSON). Of course, both cases can still be implemented (e.g., by using javascript inside an endpoint with suffix .php and by delivering XML through a resource with extension .json ), but conventions have their meaning, and developers would be confused and software agents (e.g., Web servers) may not properly handle these mismatches.

5.2 API Maturity Levels

In order to estimate the compliance of the identified APIs with the maturity levels by Richardson, we leverage on some of the above heuristics to implement composite logics representing each of the four levels of maturity. Again, we study the dataset of 18.2 million API requests and group requests by host name to study API providers rather than individual requests or APIs. Starting from the heuristics introduced earlier, we assign maturity levels to hosts as follows:

Level 0, Tunneling . As from a given URL is not possible to derive in practice whether the respective API consists of one endpoint only, we check whether the requests declare actions as query parameters ( rActionInQuery ), whether they pass resource identifiers as a parameters ( rIdInQuery ), or whether they have a resource name that suggests tunneling ( rResNameApi ).

Level 1, Resources . Here we look for APIs that use multiple endpoints that however do not yet make proper use of the semantics of HTTP. The heuristics we use here are CRUD names as resources ( rCrudResource ), problems in content negotiation ( rFormatExtension , rQueryExtension ), self-descriptiveness ( rMatchMedia ), and use of headers ( rCacheQuery ).

Level 2, HTTP methods . API providers that make use of resources and proper use of HTTP qualify for this level. However, at this level APIs don’t make yet use of hypermedia links. The heuristics used for this level include the avoidance of CRUD operations in the query params ( rCrudResource ), media types as resource extension ( rFormatExtension ), media types as query parameters ( rQueryExtension ), “cache” in query params ( rCacheQuery ), as well as the matching of media types with actual content ( rMatchMedia ).

Level 3, Hypermedia . Hypermedia means links inside resource representations to enable the client to navigate among resources. The rHypermedia heuristic helps us identify resources in this level by looking for hypermedia links inside the payload of HTTP responses.

The following pseudocode implements the logic for the identification of levels ( dNumResources is the number of individual URLs accessed through a given host, dNumMethods is the number of different HTTP methods used by the requests):

Compliance with Levels 0–2 is computed on the full dataset containing the 18.2 million requests, including both XML and JSON. Since the computation of Level 3 needs access to the actual payload of the requests, Level 3 is computed over a representative sample of the hosts complying with Level 2 (which is a prerequisite for Level 3) for which we were able to access the respective payloads. The sample consists of 1048 different requests with a confidence level of 95 % and a confidence interval of 3, along with the corresponding payloads.

The result of this analysis is illustrated in Fig.  4 (b), which reports the fractions of the studied dataset that comply with the four maturity levels. Few hosts reach Level 0; note that we explicitly focus on requests toward REST APIs and therefore excluded invocations of SOAP or XML-RPC calls by discriminating the respective media types. A significant part of the dataset complies with Level 1, yet the respective APIs do not make proper use of HTTP. The biggest part of the dataset, however, does make good use of HTTP and complies with Level 2, while only few hosts qualify for Level 3. These data indicate that the current use of REST APIs is mostly targeted at providing CRUD access to individual resources (Level 1 and 2), while full-fledged APIs that properly interlink resources and use hypermedia as the engine of state are still rare (Level 3).

Despite big steps towards resource-oriented services, there is still a large percentage of services not taking full advantage of the HTTP protocol to provide true standard interfaces. Developers should be more aware of the benefits of standard interfaces, e.g., to be compliant with the increasing number of libraries and frameworks (e.g., backbone.js, ember.js) based on RESTful principles. The limited support of hypermedia, comes as no surprise as there is no agreement on ( de facto ) standards or formats, at least not in JSON, to make the required investment by both service providers and clients worthwhile.

6 Related Work

Large scale analyses of HTTP requests have been presented in several works, but focusing mainly on quality of service [ 11 ], user profiling [ 14 ] or the general understanding of Internet traffic [ 15 ]. Analyses of RESTful design patters and anti-patterns have been the subject of recent studies [ 18 , 19 ]. Palma et al. [ 18 ] presented a heuristic-based approach for automatically detecting anti-patterns in REST APIs, namely SODA-R, that relies on service interface definitions and service invocation. The authors analyzed 12 popular REST APIs, finding anti-patterns in all of them, with more anti-patterns than patterns in services like Dropbox and Twitter. As an extension, the same authors [ 19 ] also looked at linguistic properties in 15 widely-used APIs with similar results. These studies provide insight into design patterns and tell us that even popular REST APIs have their issues. However, these works focus more on the validation of the proposed frameworks rather than on a large scale analysis of API design practices.

In contrast, in this paper we perform a large-scale analysis of REST API design best practices and of the underlying principles by studying up-to-date Mobile Internet traffic traces. Although limited by the metadata available, the large scale of the analysis presented in this paper gives us insights into the current practice that was not present in the aforementioned studies.

7 Conclusion

The work described in this paper advances the state of the art in Web engineering with three core contributions: First, to the best of our knowledge this is the first work that empirically studies how well the developers of REST APIs follow the theoretical principles and guidelines that characterize the REST architectural style. Second, the work defines a set of heuristics and metrics that allow one to measure implementation anti-patterns and API maturity levels. Third, the respective findings clearly show that, while REST APIs have irreversibly percolated into modern Web engineering practice, the gap between theory and practice is still surprisingly wide, and only very few of the analyzed APIs reach the highest level of maturity.

These findings all point into one direction: The implementation and usage of REST APIs – as well as that of Web services more in general – is still far from being a stable and consolidated discipline. On the one hand, this asks for better, principled Resource-Oriented and, in general, Service-Oriented Computing (SOC) methodologies, tools and skills [ 12 ]; pure technologies are mature enough. On the other hand, keeping in mind the ever growing strategic importance of APIs to business, this asks for better and more targeted service/API quality and usage monitoring instruments, such as proper KPIs for APIs [ 17 ].

An, X., Kunzmann, G.: Understanding mobile internet usage behavior. In: Networking Conference, IFIP 2014, pp. 1–9. IEEE (2014)

Google Scholar  

Berners-Lee, T., Fielding, R., Masinter, L.: Uniform Resource Identifier (URI): Generic syntax. Technical report (2004)

Box, D., Ehnebuske, D., Kakivaya, G., Layman, A., Mendelsohn, N., Nielsen, H.F., Thatte, S., Winer, D.: SOAP Version 1.2. W3c recommendation, W3C. http://www.w3.org/TR/soap/

Christensen, E., Curbera, F., Meredith, G., Weerawarana, S.: Web Services Description Language (WSDL) 1.1. W3c note, W3C, March 2001

Daniel, F., Matera, M.: Mashups: Concepts, Models and Architectures. Data-Centric Systems and Applications. Springer, Heidelberg (2014)

CrossRef   Google Scholar  

Fielding, R.: Architectural styles and the design of network-based software architectures. Ph.D. dissertation, University of California, Irvine (2007)

Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P., Berners-Lee, T.: Hypertext Transfer Protocol - HTTP/1.1. Technical Report RFC 2616, The Internet Society, (1999). http://www.ietf.org/rfc/rfc2616.txt

Fowler, M.: Richardson maturity model: steps toward the glory of rest (2010). http://martinfowler.com/articles/richardsonMaturityModel.html

Goland, Y., Whitehead, E., Faizi, A., Carter, S., Jensen, D.: HTTP Extensions for Distributed Authoring - WEBDAV. Rfc 2518, The Internet Society (1999). https://tools.ietf.org/html/rfc2518

Hadley, M.: Web Application Description Language. W3C member submission, Sun Microsystems (2009). http://www.w3.org/Submission/wadl/

Khirman, S., Henriksen, P.: Relationship between quality-of-service and quality-of-experience for public internet service. In: Proceedings of the 3rd Workshop on Passive and Active Measurement (2002)

Lagares Lemos, A., Daniel, F., Benatallah, B.: Web service composition: a survey of techniques and tools. ACM Comput. Surv. 48 (33), 1–41 (2015)

Mah, B., et al.: An empirical model of http network traffic. In: INFOCOM 1997, vol. 2, pp. 592–600 (1997)

Mai, T., Ajwani, D., Sala, A.: Profiling user activities with minimal traffic traces. In: Cimiano, P., Frasincar, F., Houben, G.-J., Schwabe, D. (eds.) ICWE 2015. LNCS, vol. 9114, pp. 116–133. Springer, Heidelberg (2015)

Maier, G., Schneider, F., Feldmann, A.: A first look at mobile hand-held device traffic. In: Krishnamurthy, A., Plattner, B. (eds.) PAM 2010. LNCS, vol. 6032, pp. 161–170. Springer, Heidelberg (2010)

Masse, M.: REST API design rulebook. O’Reilly Media Inc, Sebastopol (2011)

Musser, J.: KPIs for APIs. The Business of APIs Conference (2014). http://www.slideshare.net/jmusser/kpis-for-apis

Palma, F., Dubois, J., Moha, N., Guéhéneuc, Y.-G.: Detection of REST patterns and antipatterns: a heuristics-based approach. In: Franch, X., Ghose, A.K., Lewis, G.A., Bhiri, S. (eds.) ICSOC 2014. LNCS, vol. 8831, pp. 230–244. Springer, Heidelberg (2014)

Palma, F., Gonzalez-Huerta, J., Moha, N., Guéhéneuc, Y.-G., Tremblay, G.: Are RESTful APIs well-designed? detection of their linguistic (Anti)Patterns. In: Barros, A., et al. (eds.) ICSOC 2015. LNCS, vol. 9435, pp. 171–187. Springer, Heidelberg (2015). doi: 10.1007/978-3-662-48616-0_11

Pang, R., Allman, M., Bennett, M., Lee, J., Paxson, V., Tierney, B.: A first look at modern enterprise traffic. In: Proceedings of the 5th ACM SIGCOMM Conference on Internet Measurement, pp. 2–2. USENIX Association (2005)

Papazoglou, M.P.: Web Services - Principles and Technology. Prentice Hall, Upper Saddle River (2008)

Pautasso, C.: Some rest design patterns (and anti-patterns) (2009)

Pautasso, C.: Restful web services: principles, patterns, emerging technologies. In: Bouguettaya, A., Sheng, Q.Z., Daniel, F. (eds.) Web Services Foundations, pp. 31–51. Springer, New York (2014)

Bhole, Y., Popescu, A.: Measurement and analysis of HTTP traffic. J. Netw. Syst. Manage. 13 (4), 357–371 (2005)

Download references

Acknowledgement

This research has received funding from the Provincia Autonoma di Trento under the project e2Call (Enhanced Emergency Call), grant agreement number 82/13. The authors thank all partners within e2Call for their contribution.

Author information

Authors and affiliations.

University of Trento, Povo, TN, Italy

Carlos Rodríguez, Marcos Baez & Fabio Casati

Politecnico di Milano, Milan, Italy

Florian Daniel

Telecom Italia, Trento, Italy

Juan Carlos Trabucco, Luigi Canali & Gianraffaele Percannella

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Carlos Rodríguez .

Editor information

Editors and affiliations.

Dept. of Software & Computer Technology, Delft Univ. of Technology, Delft, Zuid-Holland, The Netherlands

Alessandro Bozzon

Department of Informatics, University of Fribourg, Fribourg, Switzerland

Philippe Cudre-Maroux

Faculty of Informatics, Università della Svizzera italiana (USI), Lugano, Switzerland

Cesare Pautasso

Rights and permissions

Reprints and Permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this paper

Cite this paper.

Rodríguez, C. et al. (2016). REST APIs: A Large-Scale Analysis of Compliance with Principles and Best Practices. In: Bozzon, A., Cudre-Maroux, P., Pautasso, C. (eds) Web Engineering. ICWE 2016. Lecture Notes in Computer Science(), vol 9671. Springer, Cham. https://doi.org/10.1007/978-3-319-38791-8_2

Download citation

DOI : https://doi.org/10.1007/978-3-319-38791-8_2

Published : 25 May 2016

Publisher Name : Springer, Cham

Print ISBN : 978-3-319-38790-1

Online ISBN : 978-3-319-38791-8

eBook Packages : Computer Science Computer Science (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Find a journal
• Publish with us

Search code, repositories, users, issues, pull requests...

Provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

• Notifications