No, granularity is not always related to time. Your lowest granularity will often be some kind of transaction. One of the examples Kimball uses is from a retail setting: the lowest granularity relating to product sales might be an item being scanned at the check-out. Two such transactions could happen at the same moment, so this is not a time-based granularity.
Just about anything could be the granularity of a table, but Kimball advises working to the lowest granularity as this is far more flexible - you can then slice and dice your data in more ways. You might choose to have some aggregated tables where you sum data up to Week level, or State level, or pretty much anything else (possibly for performance reasons, or to make it easier for certain users) - but these are unlikely to be your lowest granularity.
Using State as an example - you presumably have lower level information within the same hierarchy that you could analyse sales data by, like county, city, ZIP code. You also may well have data on the individual customer, the specific order reference, which shop or sales office was involved, which employees were involved in processing the order, etc. So it would be odd to choose to use State as the granularity of a fact table, unless you had some specific reason to aggregate up from a transaction fact table that was based on something like order item.
Where you often see date or time fields as the granularity of a table is in periodic snapshot facts, but again these are generally aggregated up from other, lower-granularity data sources.