1

I am trying to test https://aws.amazon.com/blogs/machine-learning/llama-2-foundation-models-from-meta-are-now-available-in-amazon-sagemaker-jumpstart/ llama2 model on sagemaker studio.

I am able to run the code in sagemkaer notebook and when I run it it gives me endpoints a well.

So I have endpoint link. Usually when I use this link with AWS credentials on my postman, it gives me response from the model.

But for Llama 2 model when I am trying to use the endpoint in postman I am getting this error

> {
>     "ErrorCode": "CLIENT_ERROR_FROM_MODEL",
>     "LogStreamArn": "arn:aws:logs:us-east-1:847137928610:log-group:/aws/sagemaker/Endpoints/meta-textgeneration-llama-2-7b-f-2023-07-26-06-06-21-772",
>     "Message": "Received client error (424) from primary with message \"{\n  \"code\":424,\n  \"message\":\"prediction failure\",\n 
> \"error\":\"Need to pass custom_attributes='accept_eula=true' as part
> of header. This means you have read and accept the
> end-user-license-agreement (EULA) of the model. EULA can be found in
> model card description or from
> https://ai.meta.com/resources/models-and-libraries/llama-downloads/.\"\n}\".
> See
> https://us-east-1.console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/sagemaker/Endpoints/meta-textgeneration-llama-2-7b-f-2023-07-26-06-06-21-772
> for more information.",
>     "OriginalMessage": "{\n  \"code\":424,\n  \"message\":\"prediction failure\",\n  \"error\":\"Need to pass
> custom_attributes='accept_eula=true' as part of header. This means you
> have read and accept the end-user-license-agreement (EULA) of the
> model. EULA can be found in model card description or from
> https://ai.meta.com/resources/models-and-libraries/llama-downloads/.\"\n}",
>     "OriginalStatusCode": 424 }

I also tried to invote sagemaker endpoint using Lambda and API Gateways following https://medium.com/@woyera/how-to-use-llama-2-with-an-api-on-aws-to-power-your-ai-apps-3e5f93314b54

But there also I am getting

{
    "message": "Internal Server Error"
}

Any suggestion or help recommended here?

navule
  • 3,212
  • 2
  • 36
  • 54
user2129623
  • 2,167
  • 3
  • 35
  • 64

3 Answers3

1

Add it as one of headers to your request, since the blog you linked mentions it:

Note that by default, accept_eula is set to false. You need to set accept_eula=true to invoke the endpoint successfully. By doing so, you accept the user license agreement and acceptable use policy as mentioned earlier. You can also download the license agreement.

Naveen Reddy Marthala
  • 2,622
  • 4
  • 35
  • 67
1

You have to send custom_attrtibutes with "accept_eula=true" in the request headers as follows when you query the deployed model endpoint or the predictor.

predictor.predict(payload, custom_attributes="accept_eula=true")

You can find sample notebook here.

navule
  • 3,212
  • 2
  • 36
  • 54
0

You have to add the following custom attribute in the POST request header:

key: X-Amzn-SageMaker-Custom-Attributes

value: accept_eula=true

vahbuna
  • 31
  • 3