Not entirely. The ancients kept time reasonably well just by being able to know the solstices and the equinoxes, the rising of certain stars in certain places, and the phases of the moon. When the earth passes through either the orbital points of solstice and equinox, we can tell precisely when that happens astronomically. The ancients went to enormous lengths to know this information and, today, their devices for doing just this are among the earliest human artifacts we have. Now we know down to the second when earth makes these four important annual passages.
I think it would make sense to standardize time worldwide. Have just one hour at one time for every place on earth. I think ultimately it would save a lot of money even it might seem odd. Anything that's globally coordinated already uses what is called UTC (Universal Coordinated Time), which is essentially Greenwich time, or Zulu Time to aviators.
What we decide to call time is arbitrary, but its passage is not.
What you're talking about is not time--it's the manner in which we've chosen to quantify and measure that thing which we call time. In other words, the Earth's revolutions and rotations are not time--they are movement, and we have chosen to use them as units of measurement (years and days, respectively). All other units of time--seconds, minutes, etc--are just arbitrary divisions of those other units (and are, in fact, the only leftover we have of Sumerian mathematics, based on a base 60 numbering system [unlike the base 10 system we use today]).
You say that the passage of time is not arbitrary, but it is. It only exists because we say it does, it only matters because we say it does. It is not a cosmic, universal constant. It is really impossible to talk about the passage of time without defining what time is, and it is impossible to define time without talking about its passing--circular logic at its best.
Time can't be defined properly because it doesn't exist, except within our minds. It is born of the human brain's need to organize and segment and compartmentalize everything it comes across. And the decision about how to name and measure this process was an arbitrary decision.
Had the course of human evolution gone in a different direction, our conception of time could very easily be quite different than it is now, or even non-existent (there have been cultures that have no concept of time as we imagine it). We can only say that "the ancients kept time reasonably well" because we are still working within the parameters that they set, and using the same methods of keeping time that they did--only they have been refined and "perfected." Just because time has been measured for thousands of years doesn't mean that it's a real thing.