Many PE lessons are failing to improve pupils' fitness, while not enough youngsters are playing competitive sport to a high level, inspectors have warned.
In a new report, Ofsted raised concerns that many schools are failing to push their sportiest pupils, or help those that are overweight.
It said in some PE lessons, there is not enough physical strenuous activity, with pupils spending too much time listening to teachers.
Overall, PE lessons are not up to scratch in around a third of primary schools and about a quarter of secondaries, said the inspectorate.
The report, based on inspections of PE in schools over the last four years, concludes that in general, the subject is "in good health", with significant investment in the last decade. But it warns that in more than a quarter of schools, PE teaching did not improve pupils' physical fitness.
Ofsted chief inspector Sir Michael Wilshaw said that PE was part of a child's entitlement to a good education, and that there are some areas for improvement.
He said: "In particular, we found there often wasn't enough physical, strenuous activity in PE lessons. Some teachers talked for too long and pupils were not provided with enough activity to enable them to learn or practise their skills. In many of the schools visited, the more able pupils were not challenged sufficiently because teachers' expectations of them were too low."
In a number of primary schools, youngsters were subjected to PE classes with warm-ups that were too short and too easy, followed by "long periods of inactivity" while the teacher talked, said the report. In secondary schools, sporty pupils were not given enough time to practise and achieve their best, the report said, and only a minority of schools played competitive sport to very high levels.
In general, "Too much teacher-talk and regular interruptions to record information and observe others performing prevented them from remaining physically active throughout lessons," said the report.
Sir Michael said that schools with good PE lessons provided an "ever increasing range of extra-curricular and traditional activities", helping pupils to achieve more. But he added: "Our report found that only a minority of schools play competitive sport to a very high level."