Setting up a FastAPI server with GPU

Using serve from ray

With an aim to make my machine learning model usable for public I could successfully hosting with CPU support only using the the following code snippet.

from fastapi import FastAPI, File, UploadFile
from ray import serve

app = FastAPI()
serve.start(detached=True)

@serve.deployment
@serve.ingress(app)
class ModelServer:
    def __init__(self):
        self.model = SomeModel()

    def action(self):
        some_code()
        return {"key": value}

    @app.post("/method")
    async def call_action(self, file…