Much may depend on what children are watching and whether they are supervised, say researchers
Spending hours watching TV or playing computer games each day does not harm young children's social development, say experts.
The Medical Research Council (MRC) team who studied more than 11,000 primary school pupils says it is wrong to link bad behaviour to TV viewing.
Although researchers found a small correlation between the two, they say other influences, such as parenting styles, most probably explain the link.
But they still say "limit screen time".
This cautionary advice is because spending lots of time in front of the TV every day might reduce how much time a child spends doing other important activities such as playing with friends and doing homework, they say.
US research suggests watching TV in early childhood can cause attention problems at the age of seven.
In the US, paediatric guidelines recommend that total screen time should be limited to less than two hours of educational, non-violent programmes per day. There are currently no formal guidelines in the UK.
For the MRC study, published in Archives of Diseases in Childhood, Dr Alison Parkes and colleagues asked UK mothers from all walks of life to give details about their child's TV viewing habits and general behaviour.
Almost two-thirds (65%) of the 11,014 five-year-olds included in the study watched TV between one and three hours a day, 15% watched more than three hours and less than 2% watched no television at all.
Watching more than three hours' TV a day at this age predicted a very small increase in "conduct" problems at the age of seven.
After their seventh birthday, these boys and girls were slightly more likely to get into fights, tell lies or be bullies than their peers, according to their mothers' reports.
Time spent playing computer games bore no such relationship.
And there was no association between TV or any screen time and other issues such as hyperactivity or problems interacting with friends.
Dr Parkes, head of the MRC's social and public health sciences unit in Glasgow, said it was wrong to blame social problems on TV.
"We found no effect with screen time for most of the behavioural and social problems that we looked at and only a very small effect indeed for conduct problems, such as fighting or bullying.
"Our work suggests that limiting the amount of time children spend in front of the TV is, in itself, unlikely to improve psychosocial adjustment."
She said interventions focusing on the family dynamic and the child were more likely to make a difference and that much may depend on what children are watching and whether they were supervised.
Sonia Livingstone, professor of social psychology, at the London School of Economics, said the findings were a "good reason to ask why some children spend so much time watching television".
Prof Annette Karmiloff-Smith, of Birkbeck, University of London, said that rather than focusing on the possible adverse effects of TV and video games, it would be better to look at what positive impact they could have on children.
Prof Hugh Perry, chair of the MRC's neurosciences and mental health board, said: "We are living in a world that is increasingly dominated by electronic entertainment, and parents are understandably concerned about the impact this might be having on their children's wellbeing and mental health.
"This important study suggests the relationship between TV and video games and health is complex and influenced by many other social and environmental factors."