Pagination, not able to view all data queried from datastore

Notebook
1

I am new to notebooks and can not get all of the data to display. When I pull the data from the original data source there are 50 rows. But when I execute from notebooks using the below, I only get 14. The data is large and I understand there is a limit, so using the max_rows is not an option. I read you need pagination which can be accomplished with the below, but still not working. Any know please?

hash_value = ‘myDataIdentifier’

while ‘LastEvaluatedKey’ in response:
response = table.query(
IndexName=‘table_key-insertTS-index’,
KeyConditionExpression = Key(‘table_key’).eq(hash_value)
,ExclusiveStartKey=response[‘LastEvaluatedKey’]
)
data = response[‘Items’]

len(temp_df.index)
temp_df = pd.DataFrame(data)
temp_df

2

TL;DR: if you really don’t care about maybe wedging your browser:

pandas.options.display.max_rows = pandas.options.display.max_columns = pandas.options.display.max_colwidth = int(1e6)

The pandas docs are excellent, once you know where to look. This particular thing is handled in options, and an example right at the top of the page has what you need.

1 Like