Introduction to gRPC-Gateway
gRPC-Gateway is a plugin for protoc. It works by reading a gRPC service definition and generating a reverse proxy server that translates a RESTful JSON API into gRPC.
This server is generated according to the custom options in your gRPC definition.
Installation and Usage
Dependencies
| Tool | Introduction | Installation |
|---|---|---|
| protobuf | Command line tool for protocol buffer compilation | http://google.github.io/proto-lens/installing-protoc.html |
| protoc-gen-go | Generates .go files from proto files | https://grpc.io/docs/languages/go/quickstart/ |
| protoc-gen-go-grpc | Generates gRPC related .go files from proto files | https://grpc.io/docs/languages/go/quickstart/ |
| protoc-gen-grpc-gateway | Generates gRPC-gateway related .go files from proto files | https://github.com/grpc-ecosystem/grpc-gateway#installation |
| protoc-gen-openapiv2 | Generates parameter files required for Swagger documentation from proto files | https://github.com/grpc-ecosystem/grpc-gateway#installation |
| buf | Protobuf management tool, optional, simplifies command line operations and protobuf file management | https://docs.buf.build/installation |
Steps
Write buf configuration
buf.gen.yaml
version: v1beta1
plugins:
- name: go
out: internal/proto
opt:
- paths=source_relative
- name: go-grpc
out: internal/proto
opt:
- paths=source_relative
- require_unimplemented_servers=false
- name: grpc-gateway
out: internal/proto
opt:
- paths=source_relative
- name: openapiv2
out: openapi
opt:
- json_names_for_fields=false
buf.yaml
version: v1beta1
name: buf.build/myworkspace/grpc
deps:
- buf.build/beta/googleapis
- buf.build/grpc-ecosystem/grpc-gateway
build:
roots:
- proto
lint:
use:
- DEFAULT
rpc_allow_google_protobuf_empty_requests: true
rpc_allow_google_protobuf_empty_responses: true
breaking:
use:
- FILE
Write proto
Unlike standard gRPC interface proto files, this includes annotations for HTTP.
- Add HTTP API information via gRPC options, such as HTTP method, path, body, additional bindings, etc. Requires importing google/api/annotations.proto
- Add extra information via protobuf message fields to implement HTTP request parameter and response body constraints, and to generate Swagger documentation. Requires importing protoc-gen-openapiv2/options/annotations.proto
demo.proto
syntax = "proto3";
package console.v1;
option go_package = "git.yourcompany.com/yourgroup/grpc-gateway-demo/proto/console/v1";
import "google/protobuf/empty.proto";
import "google/api/annotations.proto";
import "protoc-gen-openapiv2/options/annotations.proto";
// DemoService is the demo service definition
service DemoService {
rpc Hello(HelloRequest) returns (HelloResponse) {
option (google.api.http) = {
get: "/web/v1/hello-messages/{name}"
additional_bindings {
get: "/client/v1/hello-messages/{name}"
}
};
}
}
message HelloRequest {
string name = 1 [(grpc.gateway.protoc_gen_openapiv2.options.openapiv2_field) = {
description: "Name",
required: ['name'],
type: STRING,
}];
}
message HelloResponse {
string message = 1 [(grpc.gateway.protoc_gen_openapiv2.options.openapiv2_field) = {
description: "Response Message",
required: ['message'],
type: STRING,
}];
}
Generate Code
Command
grpc.sh
#!/usr/bin/env bash
buf lint
buf breaking --against .git#branch=master
buf generate
Directory Structure
demo.pb.gw.go is the file generated for gRPC-gateway, handling HTTP RESTful to gRPC requests.
grpc-gateway-demo/internal/proto
qtmf-demo/internal/proto
➜ tree
.
└── console
└── v1
├── demo.pb.go
├── demo.pb.gw.go
└── demo_grpc.pb.go
Generated Swagger Documentation
[DemoService]
GET /web/v1/hello-messages/{name}
GET /client/v1/hello-messages/{name}
Write Go RPC Service
grpc/main.go
type DemoHandler struct{}
// Hello implementation
func (s *DemoHandler) Hello(ctx context.Context, request *consoleV1.HelloRequest) (*consoleV1.HelloResponse, error) {
return &consoleV1.HelloResponse{
Message: "hello: " + request.GetName(),
}, nil
}
func main() {
lis, err := net.Listen("tcp", port)
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := grpc.NewServer()
pb.RegisterDemoServer(s, &server{})
// Register reflection service on gRPC server.
reflection.Register(s)
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
Start Service
main.go
package main
import (
"context"
"flag"
"net/http"
"github.com/golang/glog"
"github.com/grpc-ecosystem/grpc-gateway/v2/runtime"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
gw "github.com/yourorg/yourrepo/proto/gen/go/your/service/v1/your_service" // Update
)
var (
// command-line options:
// gRPC server endpoint
grpcServerEndpoint = flag.String("grpc-server-endpoint", "localhost:9090", "gRPC server endpoint")
)
func run() error {
ctx := context.Background()
ctx, cancel := context.WithCancel(ctx)
defer cancel()
// Register gRPC server endpoint
// Note: Make sure the gRPC server is running properly and accessible
mux := runtime.NewServeMux()
opts := []grpc.DialOption{grpc.WithTransportCredentials(insecure.NewCredentials())}
err := gw.RegisterYourServiceHandlerFromEndpoint(ctx, mux, *grpcServerEndpoint, opts)
if err != nil {
return err
}
// Start HTTP server (and proxy calls to gRPC server endpoint)
return http.ListenAndServe(":8081", mux)
}
func main() {
flag.Parse()
defer glog.Flush()
if err := run(); err != nil {
glog.Fatal(err)
}
}
Principle
Returning to the official diagram, it clearly shows the process of generating gRPC Service and reverse proxy from the proto file.
Source Code Analysis
gRPC-gateway is mainly divided into code generation, the generated reverse proxy, and the runtime for HTTP and gRPC protocol handling.
Code Generation
https://github.com/grpc-ecosystem/grpc-gateway/blob/master/protoc-gen-grpc-gateway/main.go#L65
func main() {
flag.Parse()
defer glog.Flush()
if *versionFlag {
fmt.Printf("Version %v, commit %v, built at %v\n", version, commit, date)
os.Exit(0)
}
protogen.Options{
ParamFunc: flag.CommandLine.Set,
}.Run(func(gen *protogen.Plugin) error {
reg := descriptor.NewRegistry()
err := applyFlags(reg)
if err != nil {
return err
}
codegenerator.SetSupportedFeaturesOnPluginGen(gen)
generator := gengateway.New(reg, *useRequestContext, *registerFuncSuffix, *allowPatchFeature, *standalone)
glog.V(1).Infof("Parsing code generator request")
if err := reg.LoadFromPlugin(gen); err != nil {
return err
}
unboundHTTPRules := reg.UnboundExternalHTTPRules()
if len(unboundHTTPRules) != 0 {
return fmt.Errorf("HTTP rules without a matching selector: %s", strings.Join(unboundHTTPRules, ", "))
}
var targets []*descriptor.File
for _, target := range gen.Request.FileToGenerate {
f, err := reg.LookupFile(target)
if err != nil {
return err
}
targets = append(targets, f)
}
files, err := generator.Generate(targets)
for _, f := range files {
glog.V(1).Infof("NewGeneratedFile %q in %s", f.GetName(), f.GoPkg)
genFile := gen.NewGeneratedFile(f.GetName(), protogen.GoImportPath(f.GoPkg.Path))
if _, err := genFile.Write([]byte(f.GetContent())); err != nil {
return err
}
}
glog.V(1).Info("Processed code generator request")
return err
})
}
Generated Reverse Proxy
// Handling HTTP requests
func RegisterGreeterHandlerServer(ctx context.Context, mux *runtime.ServeMux, server GreeterServer) error {
mux.Handle("GET", pattern_Greeter_SayHello_0, func(w http.ResponseWriter, req *http.Request, pathParams map[string]string) {
ctx, cancel := context.WithCancel(req.Context())
defer cancel()
var stream runtime.ServerTransportStream
ctx = grpc.NewContextWithServerTransportStream(ctx, &stream)
inboundMarshaler, outboundMarshaler := runtime.MarshalerForRequest(mux, req)
rctx, err := runtime.AnnotateIncomingContext(ctx, mux, req, "/grpc.gateway.examples.internal.helloworld.Greeter/SayHello", runtime.WithHTTPPathPattern("/say/{name}"))
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
resp, md, err := local_request_Greeter_SayHello_0(rctx, inboundMarshaler, server, req, pathParams)
md.HeaderMD, md.TrailerMD = metadata.Join(md.HeaderMD, stream.Header()), metadata.Join(md.TrailerMD, stream.Trailer())
ctx = runtime.NewServerMetadataContext(ctx, md)
if err != nil {
runtime.HTTPError(ctx, mux, outboundMarshaler, w, req, err)
return
}
forward_Greeter_SayHello_0(ctx, mux, outboundMarshaler, w, req, resp, mux.GetForwardResponseOptions()...)
})
// Requesting gRPC Client
func request_Greeter_SayHello_0(ctx context.Context, marshaler runtime.Marshaler, client GreeterClient, req *http.Request, pathParams map[string]string) (proto.Message, runtime.ServerMetadata, error) {
var protoReq HelloRequest
var metadata runtime.ServerMetadata
var (
val string
ok bool
err error
_ = err
)
val, ok = pathParams["name"]
if !ok {
return nil, metadata, status.Errorf(codes.InvalidArgument, "missing parameter %s", "name")
}
protoReq.Name, err = runtime.String(val)
if err != nil {
return nil, metadata, status.Errorf(codes.InvalidArgument, "type mismatch, parameter: %s, error: %v", "name", err)
}
if err := req.ParseForm(); err != nil {
return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err)
}
if err := runtime.PopulateQueryParameters(&protoReq, req.Form, filter_Greeter_SayHello_0); err != nil {
return nil, metadata, status.Errorf(codes.InvalidArgument, "%v", err)
}
msg, err := client.SayHello(ctx, &protoReq, grpc.Header(&metadata.HeaderMD), grpc.Trailer(&metadata.TrailerMD))
return msg, metadata, err
Error Mapping
Converting from gRPC status code to HTTP status code
https://github.com/grpc-ecosystem/grpc-gateway/blob/master/runtime/errors.go
// HTTPStatusFromCode converts a gRPC error code into the corresponding HTTP response status.
// See: https://github.com/googleapis/googleapis/blob/master/google/rpc/code.proto
func HTTPStatusFromCode(code codes.Code) int {
switch code {
case codes.OK:
return http.StatusOK
case codes.Canceled:
return http.StatusRequestTimeout
case codes.Unknown:
return http.StatusInternalServerError
case codes.InvalidArgument:
return http.StatusBadRequest
case codes.DeadlineExceeded:
return http.StatusGatewayTimeout
case codes.NotFound:
return http.StatusNotFound
case codes.AlreadyExists:
return http.StatusConflict
case codes.PermissionDenied:
return http.StatusForbidden
case codes.Unauthenticated:
return http.StatusUnauthorized
case codes.ResourceExhausted:
return http.StatusTooManyRequests
case codes.FailedPrecondition:
// Note, this deliberately doesn't translate to the similarly named '412 Precondition Failed' HTTP response status.
return http.StatusBadRequest
case codes.Aborted:
return http.StatusConflict
case codes.OutOfRange:
return http.StatusBadRequest
case codes.Unimplemented:
return http.StatusNotImplemented
case codes.Internal:
return http.StatusInternalServerError
case codes.Unavailable:
return http.StatusServiceUnavailable
case codes.DataLoss:
return http.StatusInternalServerError
}
grpclog.Infof("Unknown gRPC error code: %v", code)
return http.StatusInternalServerError
}
References
https://github.com/grpc-ecosystem/grpc-gateway#readme
https://grpc-ecosystem.github.io/grpc-gateway/
https://github.com/grpc/grpc-go
https://github.com/googleapis/googleapis/tree/master/google/api
https://github.com/grpc-ecosystem/grpc-gateway/tree/master/protoc-gen-openapiv2/options